[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:51:16 dist-test-slave-r0ck krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:51:16 dist-test-slave-r0ck krb5kdc[395](info): set up 2 sockets
Sep 29 15:51:16 dist-test-slave-r0ck krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 15:51:17.783367 388 mini_kdc.cc:121] Time spent starting KDC: real 2.024s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:51:17 dist-test-slave-r0ck krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:51:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:51:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:51:17.958077 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34453
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34453
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:51:18.279201 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:51:18.279471 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:51:18.279624 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:51:18.286732 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:51:18.286830 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:51:18.286876 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:51:18.286924 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:51:18.287011 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:51:18.296841 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:51:18.298161 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:51:18.299440 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:51:18.311494 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:18.311412 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:18.311467 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:51:18.312472 408 server_base.cc:1034] running on GCE node
I20260929 15:51:18.313205 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:51:18.314554 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:51:18.315740 408 hybrid_clock.cc:648] HybridClock initialized: now 1790697078315664 us; error 80 us; skew 500 ppm
I20260929 15:51:18.316072 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:51:18 dist-test-slave-r0ck krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:51:18.319061 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:51:18.320497 408 webserver.cc:460] Webserver started at http://127.0.97.62:41021/ using document root <none> and password file <none>
I20260929 15:51:18.321167 408 fs_manager.cc:359] Metadata directory not provided
I20260929 15:51:18.321305 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:51:18.321626 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:51:18.324592 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "d94719945ff6470ea89d0da15272f942"
format_stamp: "Formatted at 2026-09-29 15:51:18 on dist-test-slave-r0ck"
server_key: "f0593b06a6f3cbd49b2fce3705a36e3b"
server_key_iv: "b18025507d85db42060828328bb119cc"
server_key_version: "encryptionkey@0"
I20260929 15:51:18.325436 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "d94719945ff6470ea89d0da15272f942"
format_stamp: "Formatted at 2026-09-29 15:51:18 on dist-test-slave-r0ck"
server_key: "f0593b06a6f3cbd49b2fce3705a36e3b"
server_key_iv: "b18025507d85db42060828328bb119cc"
server_key_version: "encryptionkey@0"
I20260929 15:51:18.330892 408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 15:51:18.334252 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:18.335606 408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 15:51:18.335881 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "d94719945ff6470ea89d0da15272f942"
format_stamp: "Formatted at 2026-09-29 15:51:18 on dist-test-slave-r0ck"
server_key: "f0593b06a6f3cbd49b2fce3705a36e3b"
server_key_iv: "b18025507d85db42060828328bb119cc"
server_key_version: "encryptionkey@0"
I20260929 15:51:18.336071 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:51:18.356834 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:51:18.360620 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:51:18.360895 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:51:18.378986 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34453
I20260929 15:51:18.378996 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34453 every 8 connection(s)
I20260929 15:51:18.380344 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:51:18.383404 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 408
I20260929 15:51:18.383689 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:51:18.384459 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:18.385592 388 external_mini_cluster.cc:1428] Setting key da73112c8cd9e1feb105e41d2f894411
I20260929 15:51:18.394487 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d94719945ff6470ea89d0da15272f942: Bootstrap starting.
I20260929 15:51:18.397197 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d94719945ff6470ea89d0da15272f942: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:18.398159 475 log.cc:824] T 00000000000000000000000000000000 P d94719945ff6470ea89d0da15272f942: Log is configured to *not* fsync() on all Append() calls
I20260929 15:51:18.401273 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d94719945ff6470ea89d0da15272f942: No bootstrap required, opened a new log
I20260929 15:51:18.410213 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P d94719945ff6470ea89d0da15272f942 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d94719945ff6470ea89d0da15272f942" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34453 } }
I20260929 15:51:18.410658 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P d94719945ff6470ea89d0da15272f942 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:18.410776 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P d94719945ff6470ea89d0da15272f942 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d94719945ff6470ea89d0da15272f942, State: Initialized, Role: FOLLOWER
I20260929 15:51:18.411466 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P d94719945ff6470ea89d0da15272f942 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d94719945ff6470ea89d0da15272f942" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34453 } }
I20260929 15:51:18.411727 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P d94719945ff6470ea89d0da15272f942 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:51:18.411859 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P d94719945ff6470ea89d0da15272f942 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:51:18.412052 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d94719945ff6470ea89d0da15272f942 [term 0 FOLLOWER]: Advancing to term 1
Sep 29 15:51:18 dist-test-slave-r0ck krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:51:18.414258 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P d94719945ff6470ea89d0da15272f942 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d94719945ff6470ea89d0da15272f942" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34453 } }
I20260929 15:51:18.414712 475 leader_election.cc:302] T 00000000000000000000000000000000 P d94719945ff6470ea89d0da15272f942 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d94719945ff6470ea89d0da15272f942; no voters:
I20260929 15:51:18.415849 475 leader_election.cc:288] T 00000000000000000000000000000000 P d94719945ff6470ea89d0da15272f942 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:51:18.415995 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d94719945ff6470ea89d0da15272f942 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:51:18.417307 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P d94719945ff6470ea89d0da15272f942 [term 1 LEADER]: Becoming Leader. State: Replica: d94719945ff6470ea89d0da15272f942, State: Running, Role: LEADER
I20260929 15:51:18.417939 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P d94719945ff6470ea89d0da15272f942 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d94719945ff6470ea89d0da15272f942" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34453 } }
I20260929 15:51:18.418253 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P d94719945ff6470ea89d0da15272f942 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:51:18.420774 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P d94719945ff6470ea89d0da15272f942 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d94719945ff6470ea89d0da15272f942. Latest consensus state: current_term: 1 leader_uuid: "d94719945ff6470ea89d0da15272f942" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d94719945ff6470ea89d0da15272f942" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34453 } } }
I20260929 15:51:18.420871 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P d94719945ff6470ea89d0da15272f942 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d94719945ff6470ea89d0da15272f942" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d94719945ff6470ea89d0da15272f942" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34453 } } }
I20260929 15:51:18.421160 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P d94719945ff6470ea89d0da15272f942 [sys.catalog]: This master's current role is: LEADER
I20260929 15:51:18.421238 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P d94719945ff6470ea89d0da15272f942 [sys.catalog]: This master's current role is: LEADER
I20260929 15:51:18.422132 485 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:51:18.428380 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:18.389254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40854 (local address 127.0.97.62:34453)
0929 15:51:18.390109 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:18.390124 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:18.391392 (+ 1268us) server_negotiation.cc:407] Connection header received
0929 15:51:18.400676 (+ 9284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:18.400699 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:18.401119 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:18.401624 (+ 505us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:18.404079 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:18.406567 (+ 2488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:18.408152 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:18.409243 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:18.409445 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:18.415813 (+ 6368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:18.415821 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:18.415838 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:18.415879 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:18.419757 (+ 3878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:18.420874 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:18.420882 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:18.420890 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:18.421131 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:18.421775 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:18.421779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:18.421782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:18.422462 (+ 680us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:18.425638 (+ 3176us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:18.425968 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 15:51:18.426472 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":361,"threads_started":1}
I20260929 15:51:18.434975 485 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:51:18.451979 485 catalog_manager.cc:1347] Generated new cluster ID: bab4e873d6934141a460eb1e9d624d11
I20260929 15:51:18.452117 485 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:51:18.465605 485 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:51:18.467790 485 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:51:18.481972 485 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d94719945ff6470ea89d0da15272f942: Generated new TSK 0
I20260929 15:51:18.482798 485 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 15:51:18.534780 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34453
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:51:18.915910 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:51:18.916199 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:51:18.916304 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:51:18.924729 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:51:18.924837 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:51:18.924950 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:51:18.935000 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34453
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:51:18.936306 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:51:18.937702 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:51:18.951359 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:18.951318 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:18.951329 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:51:18.957547 501 server_base.cc:1034] running on GCE node
I20260929 15:51:18.958088 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:51:18.958969 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:51:18.960176 501 hybrid_clock.cc:648] HybridClock initialized: now 1790697078960126 us; error 58 us; skew 500 ppm
I20260929 15:51:18.960484 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:51:18 dist-test-slave-r0ck krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:51:18.963653 501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:51:18.965209 501 webserver.cc:460] Webserver started at http://127.0.97.1:45513/ using document root <none> and password file <none>
I20260929 15:51:18.965951 501 fs_manager.cc:359] Metadata directory not provided
I20260929 15:51:18.966079 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:51:18.966394 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:51:18.969208 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-0/data/instance:
uuid: "769be527b8bb462899eeada39ff9de85"
format_stamp: "Formatted at 2026-09-29 15:51:18 on dist-test-slave-r0ck"
server_key: "e886692d2c17aaa61212f45e1e76a840"
server_key_iv: "d522c8fa12e5c464b40dd09beb3096d6"
server_key_version: "encryptionkey@0"
I20260929 15:51:18.970041 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "769be527b8bb462899eeada39ff9de85"
format_stamp: "Formatted at 2026-09-29 15:51:18 on dist-test-slave-r0ck"
server_key: "e886692d2c17aaa61212f45e1e76a840"
server_key_iv: "d522c8fa12e5c464b40dd09beb3096d6"
server_key_version: "encryptionkey@0"
I20260929 15:51:18.975397 501 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260929 15:51:18.978730 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:18.979949 501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:51:18.980124 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-0/wal
uuid: "769be527b8bb462899eeada39ff9de85"
format_stamp: "Formatted at 2026-09-29 15:51:18 on dist-test-slave-r0ck"
server_key: "e886692d2c17aaa61212f45e1e76a840"
server_key_iv: "d522c8fa12e5c464b40dd09beb3096d6"
server_key_version: "encryptionkey@0"
I20260929 15:51:18.980314 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:51:18.996714 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:51:19.000837 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:51:19.001129 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:51:19.002048 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:51:19.003309 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:51:19.003413 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:19.003566 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:51:19.003636 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:19.037801 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46227
I20260929 15:51:19.037844 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46227 every 8 connection(s)
I20260929 15:51:19.039048 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:51:19.042399 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 501
I20260929 15:51:19.042609 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:51:19.043427 388 external_mini_cluster.cc:1428] Setting key c2ac4307063d808c3838de74345c826a
Sep 29 15:51:19 dist-test-slave-r0ck krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 15:51:19.058565 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:19.041926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48089 (local address 127.0.97.62:34453)
0929 15:51:19.042354 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:19.042358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:19.043499 (+ 1141us) server_negotiation.cc:407] Connection header received
0929 15:51:19.045325 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:19.045330 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:19.045451 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:19.045565 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:19.047027 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:19.047731 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:19.049281 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:19.049742 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:19.049885 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:19.053151 (+ 3266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:19.053157 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:19.053160 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:19.053189 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:19.055710 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:19.056381 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:19.056387 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:19.056392 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:19.056509 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:19.056926 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:19.056933 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:19.056936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:19.057185 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:19.057325 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:19.058004 (+ 679us) server_negotiation.cc:299] Negotiation successful
0929 15:51:19.058212 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":237,"threads_started":1}
I20260929 15:51:19.059484 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:19.042583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34453 (local address 127.0.97.1:48089)
0929 15:51:19.043338 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:19.043383 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:19.045067 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:19.045806 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:19.045815 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:19.046312 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:51:19.046860 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:19.046876 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:19.047895 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:19.047900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:19.049131 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:19.049143 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:19.049977 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:19.049986 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:19.050220 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:19.050901 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:51:19.050927 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:51:19.052945 (+ 2018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:51:19.055847 (+ 2902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:19.055856 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:19.055873 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:19.056266 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:19.056624 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:19.056629 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:19.056632 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:19.056814 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:19.057316 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:51:19.057323 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:51:19.057647 (+ 324us) client_negotiation.cc:769] Sending connection context
0929 15:51:19.057908 (+ 261us) client_negotiation.cc:240] Negotiation successful
0929 15:51:19.058402 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":317,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 15:51:19.061053 629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34453
I20260929 15:51:19.061388 629 heartbeater.cc:464] Registering TS with master...
I20260929 15:51:19.062068 629 heartbeater.cc:511] Master 127.0.97.62:34453 requested a full tablet report, sending...
I20260929 15:51:19.064153 440 ts_manager.cc:194] Registered new tserver with Master: 769be527b8bb462899eeada39ff9de85 (127.0.97.1:46227)
I20260929 15:51:19.066426 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48089
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 15:51:19.081190 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34453
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:51:19.399786 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:51:19.400044 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:51:19.400205 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:51:19.407740 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:51:19.407835 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:51:19.408048 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:51:19.417577 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34453
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:51:19.418885 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:51:19.420260 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:51:19.433401 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:19.433368 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:19.433444 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:51:19.434227 635 server_base.cc:1034] running on GCE node
I20260929 15:51:19.434794 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:51:19.435696 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:51:19.436923 635 hybrid_clock.cc:648] HybridClock initialized: now 1790697079436872 us; error 56 us; skew 500 ppm
I20260929 15:51:19.437234 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:51:19 dist-test-slave-r0ck krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:51:19.440248 635 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:51:19.441733 635 webserver.cc:460] Webserver started at http://127.0.97.2:41217/ using document root <none> and password file <none>
I20260929 15:51:19.442389 635 fs_manager.cc:359] Metadata directory not provided
I20260929 15:51:19.442528 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:51:19.442821 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:51:19.445644 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-1/data/instance:
uuid: "13eea3cb773e45c3b5b990279e3ac309"
format_stamp: "Formatted at 2026-09-29 15:51:19 on dist-test-slave-r0ck"
server_key: "bd85a035cafefe82e753318724339f96"
server_key_iv: "3630a9fa237a4f0bb57615f78de1c6a1"
server_key_version: "encryptionkey@0"
I20260929 15:51:19.446415 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "13eea3cb773e45c3b5b990279e3ac309"
format_stamp: "Formatted at 2026-09-29 15:51:19 on dist-test-slave-r0ck"
server_key: "bd85a035cafefe82e753318724339f96"
server_key_iv: "3630a9fa237a4f0bb57615f78de1c6a1"
server_key_version: "encryptionkey@0"
I20260929 15:51:19.451707 635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 15:51:19.454877 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:19.456073 635 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 15:51:19.456219 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-1/wal
uuid: "13eea3cb773e45c3b5b990279e3ac309"
format_stamp: "Formatted at 2026-09-29 15:51:19 on dist-test-slave-r0ck"
server_key: "bd85a035cafefe82e753318724339f96"
server_key_iv: "3630a9fa237a4f0bb57615f78de1c6a1"
server_key_version: "encryptionkey@0"
I20260929 15:51:19.456415 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:51:19.467757 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:51:19.471405 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:51:19.471678 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:51:19.472909 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:51:19.474336 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:51:19.474417 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:19.474545 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:51:19.474603 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:19.509683 635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44903
I20260929 15:51:19.509714 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44903 every 8 connection(s)
I20260929 15:51:19.510892 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:51:19.515426 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 635
I20260929 15:51:19.515609 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:51:19.516014 388 external_mini_cluster.cc:1428] Setting key 97af8a1fe0d4d4a8cd791bad0e19b5bc
Sep 29 15:51:19 dist-test-slave-r0ck krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 15:51:19.532636 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:19.513260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48335 (local address 127.0.97.62:34453)
0929 15:51:19.513488 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:19.513494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:19.514623 (+ 1129us) server_negotiation.cc:407] Connection header received
0929 15:51:19.516462 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:19.516468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:19.516554 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:19.516653 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:19.518842 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:19.520053 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:19.521594 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:19.522281 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:19.522381 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:19.526376 (+ 3995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:19.526381 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:19.526384 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:19.526419 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:19.529613 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:19.530293 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:19.530299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:19.530303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:19.530462 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:19.530859 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:19.530863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:19.530868 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:19.531219 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:19.531386 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:19.531976 (+ 590us) server_negotiation.cc:299] Negotiation successful
0929 15:51:19.532206 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 15:51:19.533417 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:19.513825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34453 (local address 127.0.97.2:48335)
0929 15:51:19.514475 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:19.514512 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:19.516209 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:19.516913 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:19.516921 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:19.517490 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:51:19.518672 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:19.518692 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:19.520193 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:19.520197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:19.521459 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:19.521470 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:19.522513 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:19.522520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:19.522837 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:19.523566 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:51:19.523599 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:51:19.526155 (+ 2556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:51:19.529746 (+ 3591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:19.529752 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:19.529765 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:19.530162 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:19.530578 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:19.530581 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:19.530583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:19.530740 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:19.531331 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:51:19.531336 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:51:19.531624 (+ 288us) client_negotiation.cc:769] Sending connection context
0929 15:51:19.531830 (+ 206us) client_negotiation.cc:240] Negotiation successful
0929 15:51:19.532288 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":248,"threads_started":1}
I20260929 15:51:19.534778 763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34453
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 15:51:19.535123 763 heartbeater.cc:464] Registering TS with master...
I20260929 15:51:19.535981 763 heartbeater.cc:511] Master 127.0.97.62:34453 requested a full tablet report, sending...
I20260929 15:51:19.537513 440 ts_manager.cc:194] Registered new tserver with Master: 13eea3cb773e45c3b5b990279e3ac309 (127.0.97.2:44903)
I20260929 15:51:19.538280 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:48335
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 15:51:19.556550 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34453
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:51:19.874300 768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:51:19.874615 768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:51:19.874790 768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:51:19.882320 768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:51:19.882467 768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:51:19.882645 768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:51:19.892144 768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34453
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:51:19.893402 768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:51:19.894714 768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:51:19.907706 773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:19.907670 776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:19.907742 774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:51:19.909029 768 server_base.cc:1034] running on GCE node
I20260929 15:51:19.909636 768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:51:19.910480 768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:51:19.911674 768 hybrid_clock.cc:648] HybridClock initialized: now 1790697079911641 us; error 30 us; skew 500 ppm
I20260929 15:51:19.911923 768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:51:19 dist-test-slave-r0ck krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:51:19.915040 768 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:51:19.916810 768 webserver.cc:460] Webserver started at http://127.0.97.3:35075/ using document root <none> and password file <none>
I20260929 15:51:19.917508 768 fs_manager.cc:359] Metadata directory not provided
I20260929 15:51:19.917644 768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:51:19.917944 768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:51:19.920827 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-2/data/instance:
uuid: "d7c8372783764943b7edeea3e1891319"
format_stamp: "Formatted at 2026-09-29 15:51:19 on dist-test-slave-r0ck"
server_key: "d4b005cf27cc9619e889ff544532e6a4"
server_key_iv: "e4a479a512cd3dc3070dab191912f596"
server_key_version: "encryptionkey@0"
I20260929 15:51:19.921633 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d7c8372783764943b7edeea3e1891319"
format_stamp: "Formatted at 2026-09-29 15:51:19 on dist-test-slave-r0ck"
server_key: "d4b005cf27cc9619e889ff544532e6a4"
server_key_iv: "e4a479a512cd3dc3070dab191912f596"
server_key_version: "encryptionkey@0"
I20260929 15:51:19.926419 768 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 15:51:19.929680 783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:19.930967 768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 15:51:19.931133 768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-2/wal
uuid: "d7c8372783764943b7edeea3e1891319"
format_stamp: "Formatted at 2026-09-29 15:51:19 on dist-test-slave-r0ck"
server_key: "d4b005cf27cc9619e889ff544532e6a4"
server_key_iv: "e4a479a512cd3dc3070dab191912f596"
server_key_version: "encryptionkey@0"
I20260929 15:51:19.931324 768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:51:19.950450 768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:51:19.955165 768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:51:19.955461 768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:51:19.956355 768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:51:19.957638 768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:51:19.957731 768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:19.957863 768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:51:19.957931 768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:19.992014 768 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46435
I20260929 15:51:19.992064 895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46435 every 8 connection(s)
I20260929 15:51:19.993253 768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:51:20.001406 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 768
I20260929 15:51:20.001602 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697075657932-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:51:20.002125 388 external_mini_cluster.cc:1428] Setting key fe9a2fe50de6bc33c2a3d57e6f18cc8e
Sep 29 15:51:20 dist-test-slave-r0ck krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:51:20.014837 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:19.995684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45309 (local address 127.0.97.62:34453)
0929 15:51:19.995891 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:19.995897 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:19.997008 (+ 1111us) server_negotiation.cc:407] Connection header received
0929 15:51:19.999138 (+ 2130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:19.999142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:19.999207 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:19.999294 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:20.001391 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:20.002576 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:20.004465 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:20.005096 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:20.005256 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:20.009111 (+ 3855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:20.009116 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:20.009119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:20.009157 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:20.011769 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:20.012483 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:20.012490 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:20.012493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:20.012644 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:20.013074 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:20.013081 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:20.013085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:20.013400 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:20.013556 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:20.014261 (+ 705us) server_negotiation.cc:299] Negotiation successful
0929 15:51:20.014488 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 15:51:20.015843 898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:19.996192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34453 (local address 127.0.97.3:45309)
0929 15:51:19.996850 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:19.996888 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:19.998899 (+ 2011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:19.999562 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:19.999570 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:20.000595 (+ 1025us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:51:20.001168 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:20.001186 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:20.002757 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:20.002762 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:20.004306 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:20.004324 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:20.005604 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:20.005611 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:20.005835 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:20.006573 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:51:20.006701 (+ 128us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:51:20.008900 (+ 2199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:51:20.011901 (+ 3001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:20.011908 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:20.011928 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:20.012334 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:20.012759 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:20.012765 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:20.012769 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:20.012958 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:20.013535 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:51:20.013543 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:51:20.013875 (+ 332us) client_negotiation.cc:769] Sending connection context
0929 15:51:20.014148 (+ 273us) client_negotiation.cc:240] Negotiation successful
0929 15:51:20.014678 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":263,"threads_started":1}
I20260929 15:51:20.017405 896 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34453
I20260929 15:51:20.017764 896 heartbeater.cc:464] Registering TS with master...
I20260929 15:51:20.018702 896 heartbeater.cc:511] Master 127.0.97.62:34453 requested a full tablet report, sending...
I20260929 15:51:20.020062 440 ts_manager.cc:194] Registered new tserver with Master: d7c8372783764943b7edeea3e1891319 (127.0.97.3:46435)
I20260929 15:51:20.020807 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:45309
I20260929 15:51:20.030822 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 15:51:20 dist-test-slave-r0ck krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697080, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260929 15:51:20.070142 629 heartbeater.cc:503] Master 127.0.97.62:34453 was elected leader, sending a full tablet report...
Sep 29 15:51:20 dist-test-slave-r0ck krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697080, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 15:51:20.077822 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:20.062054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54098 (local address 127.0.97.1:46227)
0929 15:51:20.062710 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:20.062717 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:20.062743 (+ 26us) server_negotiation.cc:407] Connection header received
0929 15:51:20.062857 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:20.062864 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:20.063068 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:20.063202 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:20.064536 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:20.065595 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:20.066862 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:20.067402 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:20.067500 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:20.072070 (+ 4570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:20.072081 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:20.072096 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:20.072139 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:20.075085 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:20.075749 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:20.075757 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:20.075759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:20.075871 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:20.076272 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:20.076278 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:20.076282 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:20.076612 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:20.076914 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:20.077199 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 15:51:20.077477 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":362,"threads_started":1}
W20260929 15:51:20.079048 544 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54098
Sep 29 15:51:20 dist-test-slave-r0ck krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 15:51:20 dist-test-slave-r0ck krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 15:51:20.121035 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:20.107379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54112 (local address 127.0.97.1:46227)
0929 15:51:20.107608 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:20.107616 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:20.107877 (+ 261us) server_negotiation.cc:407] Connection header received
0929 15:51:20.108152 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:20.108157 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:20.108218 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:20.108320 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:20.109667 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:20.110347 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:20.111558 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:20.111974 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:20.112106 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:20.115725 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:20.115729 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:20.115733 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:20.115764 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:20.118811 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:20.119587 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:20.119591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:20.119594 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:20.119748 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:20.120045 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:20.120048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:20.120050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:20.120264 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:20.120422 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:20.120662 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 15:51:20.120813 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 15:51:20.122454 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 501
I20260929 15:51:20.134790 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:20.123488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54120 (local address 127.0.97.1:46227)
0929 15:51:20.123691 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:20.123696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:20.123887 (+ 191us) server_negotiation.cc:407] Connection header received
0929 15:51:20.124094 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:20.124099 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:20.124150 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:20.124244 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:20.125326 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:20.126030 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:20.127233 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:20.127633 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:20.127815 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:20.129958 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:20.129963 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:20.129966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:20.129992 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:20.132238 (+ 2246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:20.132907 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:20.132915 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:20.132918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:20.133053 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:20.133419 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:20.133424 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:20.133428 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:20.133726 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:20.133953 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:20.134234 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 15:51:20.134425 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 15:51:20.135493 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:51:20.216992 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 501
I20260929 15:51:20.233107 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 635
Sep 29 15:51:20 dist-test-slave-r0ck krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 15:51:20.248301 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:20.234016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36732 (local address 127.0.97.2:44903)
0929 15:51:20.234579 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:20.234586 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:20.234613 (+ 27us) server_negotiation.cc:407] Connection header received
0929 15:51:20.234671 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:20.234677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:20.234925 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:20.235114 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:20.236239 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:20.237251 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:20.238440 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:20.238953 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:20.239050 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:20.243119 (+ 4069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:20.243129 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:20.243144 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:20.243176 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:20.246019 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:20.246574 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:20.246581 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:20.246583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:20.246689 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:20.247013 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:20.247016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:20.247018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:20.247320 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:20.247561 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:20.247744 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 15:51:20.247920 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":313,"threads_started":1}
I20260929 15:51:20.249686 758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:51:20.332121 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 635
I20260929 15:51:20.354857 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 768
Sep 29 15:51:20 dist-test-slave-r0ck krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 15:51:20.371272 912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:20.355902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39498 (local address 127.0.97.3:46435)
0929 15:51:20.356496 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:20.356502 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:20.356529 (+ 27us) server_negotiation.cc:407] Connection header received
0929 15:51:20.356587 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:20.356593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:20.356811 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:20.356933 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:20.358228 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:20.359264 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:20.360565 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:20.361095 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:20.361187 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:20.365641 (+ 4454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:20.365655 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:20.365674 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:20.365726 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:20.368634 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:20.369285 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:20.369293 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:20.369295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:20.369479 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:20.369877 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:20.369881 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:20.369883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:20.370216 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:20.370440 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:20.370694 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 15:51:20.370884 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":301,"threads_started":1}
I20260929 15:51:20.372596 891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:51:20.458438 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 768
I20260929 15:51:20.474231 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 408
I20260929 15:51:20.475452 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:51:20.576375 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 408
2026-09-29T15:51:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4870 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:51:20 dist-test-slave-r0ck krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:51:20 dist-test-slave-r0ck krb5kdc[917](info): set up 2 sockets
Sep 29 15:51:20 dist-test-slave-r0ck krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260929 15:51:21.665138 388 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:51:21 dist-test-slave-r0ck krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:51:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:51:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:51:21.837031 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46631
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46631
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:51:22.143801 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:51:22.144076 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:51:22.144224 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:51:22.151217 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:51:22.151312 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:51:22.151413 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:51:22.151503 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:51:22.151577 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:51:22.160843 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:51:22.162204 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:51:22.163461 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:51:22.174822 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:22.174736 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:22.174777 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:51:22.175599 930 server_base.cc:1034] running on GCE node
I20260929 15:51:22.176352 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:51:22.177773 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:51:22.178988 930 hybrid_clock.cc:648] HybridClock initialized: now 1790697082178925 us; error 55 us; skew 500 ppm
I20260929 15:51:22.179325 930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:51:22 dist-test-slave-r0ck krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:51:22.186617 930 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:51:22.188026 930 webserver.cc:460] Webserver started at http://127.0.97.62:35505/ using document root <none> and password file <none>
I20260929 15:51:22.188644 930 fs_manager.cc:359] Metadata directory not provided
I20260929 15:51:22.188735 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:51:22.188998 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:51:22.191805 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "58654064a60d490ea5430c7513dc2c11"
format_stamp: "Formatted at 2026-09-29 15:51:22 on dist-test-slave-r0ck"
server_key: "7ccf59a3ff63022b4b7c09fc1c6f09a8"
server_key_iv: "1360796ea18c0d4b1ec7bf7b42c60429"
server_key_version: "encryptionkey@0"
I20260929 15:51:22.192469 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "58654064a60d490ea5430c7513dc2c11"
format_stamp: "Formatted at 2026-09-29 15:51:22 on dist-test-slave-r0ck"
server_key: "7ccf59a3ff63022b4b7c09fc1c6f09a8"
server_key_iv: "1360796ea18c0d4b1ec7bf7b42c60429"
server_key_version: "encryptionkey@0"
I20260929 15:51:22.197407 930 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260929 15:51:22.200635 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:22.201902 930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:51:22.202071 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "58654064a60d490ea5430c7513dc2c11"
format_stamp: "Formatted at 2026-09-29 15:51:22 on dist-test-slave-r0ck"
server_key: "7ccf59a3ff63022b4b7c09fc1c6f09a8"
server_key_iv: "1360796ea18c0d4b1ec7bf7b42c60429"
server_key_version: "encryptionkey@0"
I20260929 15:51:22.202251 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:51:22.217741 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:51:22.221648 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:51:22.221923 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:51:22.239749 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46631
I20260929 15:51:22.239781 996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46631 every 8 connection(s)
I20260929 15:51:22.241129 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:51:22.241534 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 930
I20260929 15:51:22.241752 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:51:22.242585 388 external_mini_cluster.cc:1428] Setting key 56e57389d5492801615623d636452382
I20260929 15:51:22.245468 997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 15:51:22 dist-test-slave-r0ck krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:51:22.258371 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58654064a60d490ea5430c7513dc2c11: Bootstrap starting.
I20260929 15:51:22.262368 997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58654064a60d490ea5430c7513dc2c11: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:22.263646 997 log.cc:824] T 00000000000000000000000000000000 P 58654064a60d490ea5430c7513dc2c11: Log is configured to *not* fsync() on all Append() calls
I20260929 15:51:22.265996 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:22.244540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33632 (local address 127.0.97.62:46631)
0929 15:51:22.245329 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:22.245387 (+ 58us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:22.245437 (+ 50us) server_negotiation.cc:407] Connection header received
0929 15:51:22.247056 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:22.247076 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:22.247440 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:22.247762 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:22.249513 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:22.250859 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:22.252560 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:22.253185 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:22.253310 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:22.257835 (+ 4525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:22.257842 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:22.257855 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:22.257897 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:22.261580 (+ 3683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:22.262182 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:22.262193 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:22.262203 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:22.262409 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:22.262815 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:22.262822 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:22.262825 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:22.263484 (+ 659us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:22.263770 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:22.264286 (+ 516us) server_negotiation.cc:299] Negotiation successful
0929 15:51:22.264731 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":366,"threads_started":1}
I20260929 15:51:22.266954 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58654064a60d490ea5430c7513dc2c11: No bootstrap required, opened a new log
I20260929 15:51:22.274152 997 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58654064a60d490ea5430c7513dc2c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58654064a60d490ea5430c7513dc2c11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46631 } }
I20260929 15:51:22.274557 997 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58654064a60d490ea5430c7513dc2c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:22.274673 997 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58654064a60d490ea5430c7513dc2c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58654064a60d490ea5430c7513dc2c11, State: Initialized, Role: FOLLOWER
I20260929 15:51:22.275287 997 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58654064a60d490ea5430c7513dc2c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58654064a60d490ea5430c7513dc2c11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46631 } }
I20260929 15:51:22.275525 997 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58654064a60d490ea5430c7513dc2c11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:51:22.275665 997 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58654064a60d490ea5430c7513dc2c11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:51:22.275848 997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58654064a60d490ea5430c7513dc2c11 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:22.277985 997 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58654064a60d490ea5430c7513dc2c11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58654064a60d490ea5430c7513dc2c11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46631 } }
I20260929 15:51:22.278481 997 leader_election.cc:302] T 00000000000000000000000000000000 P 58654064a60d490ea5430c7513dc2c11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58654064a60d490ea5430c7513dc2c11; no voters:
I20260929 15:51:22.279120 997 leader_election.cc:288] T 00000000000000000000000000000000 P 58654064a60d490ea5430c7513dc2c11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:51:22.279242 1002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58654064a60d490ea5430c7513dc2c11 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:51:22.279945 1002 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58654064a60d490ea5430c7513dc2c11 [term 1 LEADER]: Becoming Leader. State: Replica: 58654064a60d490ea5430c7513dc2c11, State: Running, Role: LEADER
I20260929 15:51:22.280465 1002 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58654064a60d490ea5430c7513dc2c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58654064a60d490ea5430c7513dc2c11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46631 } }
I20260929 15:51:22.281102 997 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58654064a60d490ea5430c7513dc2c11 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:51:22.283484 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58654064a60d490ea5430c7513dc2c11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58654064a60d490ea5430c7513dc2c11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58654064a60d490ea5430c7513dc2c11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46631 } } }
I20260929 15:51:22.283473 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58654064a60d490ea5430c7513dc2c11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58654064a60d490ea5430c7513dc2c11. Latest consensus state: current_term: 1 leader_uuid: "58654064a60d490ea5430c7513dc2c11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58654064a60d490ea5430c7513dc2c11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46631 } } }
I20260929 15:51:22.284130 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58654064a60d490ea5430c7513dc2c11 [sys.catalog]: This master's current role is: LEADER
I20260929 15:51:22.284817 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58654064a60d490ea5430c7513dc2c11 [sys.catalog]: This master's current role is: LEADER
I20260929 15:51:22.289099 1009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:51:22.294391 1009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:51:22.302889 1009 catalog_manager.cc:1347] Generated new cluster ID: 3ef52abd92f34c0bb67c2736a7591c35
I20260929 15:51:22.302987 1009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:51:22.318086 1009 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:51:22.319839 1009 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:51:22.329203 1009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58654064a60d490ea5430c7513dc2c11: Generated new TSK 0
I20260929 15:51:22.330201 1009 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 15:51:22.369625 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46631
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:51:22.683617 1023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:51:22.683885 1023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:51:22.684057 1023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:51:22.691327 1023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:51:22.691424 1023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:51:22.691591 1023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:51:22.701020 1023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46631
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:51:22.702306 1023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:51:22.703599 1023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:51:22.716535 1031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:22.716617 1028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:22.716614 1029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:51:22.717540 1023 server_base.cc:1034] running on GCE node
I20260929 15:51:22.718144 1023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:51:22.718992 1023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:51:22.720196 1023 hybrid_clock.cc:648] HybridClock initialized: now 1790697082720144 us; error 40 us; skew 500 ppm
I20260929 15:51:22.720518 1023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:51:22 dist-test-slave-r0ck krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:51:22.723578 1023 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:51:22.725008 1023 webserver.cc:460] Webserver started at http://127.0.97.1:33493/ using document root <none> and password file <none>
I20260929 15:51:22.725756 1023 fs_manager.cc:359] Metadata directory not provided
I20260929 15:51:22.725899 1023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:51:22.726207 1023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:51:22.729027 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-0/data/instance:
uuid: "308c40d919db40f1a6de8dbd8a5ae524"
format_stamp: "Formatted at 2026-09-29 15:51:22 on dist-test-slave-r0ck"
server_key: "416fee817415d178e1c5229506634193"
server_key_iv: "eb51321526823b33081140e8fbd081f8"
server_key_version: "encryptionkey@0"
I20260929 15:51:22.729820 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "308c40d919db40f1a6de8dbd8a5ae524"
format_stamp: "Formatted at 2026-09-29 15:51:22 on dist-test-slave-r0ck"
server_key: "416fee817415d178e1c5229506634193"
server_key_iv: "eb51321526823b33081140e8fbd081f8"
server_key_version: "encryptionkey@0"
I20260929 15:51:22.734707 1023 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 15:51:22.738020 1038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:22.739243 1023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 15:51:22.739415 1023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-0/wal
uuid: "308c40d919db40f1a6de8dbd8a5ae524"
format_stamp: "Formatted at 2026-09-29 15:51:22 on dist-test-slave-r0ck"
server_key: "416fee817415d178e1c5229506634193"
server_key_iv: "eb51321526823b33081140e8fbd081f8"
server_key_version: "encryptionkey@0"
I20260929 15:51:22.739616 1023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:51:22.766760 1023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:51:22.771173 1023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:51:22.771478 1023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:51:22.772429 1023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:51:22.773772 1023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:51:22.773842 1023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:22.773968 1023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:51:22.774031 1023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:22.806864 1023 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43241
I20260929 15:51:22.806893 1150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43241 every 8 connection(s)
I20260929 15:51:22.808099 1023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:51:22.816461 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 1023
I20260929 15:51:22.816685 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:51:22.817272 388 external_mini_cluster.cc:1428] Setting key 6b45c4ab5e3ffb52cbef08bf2c496bb9
Sep 29 15:51:22 dist-test-slave-r0ck krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:51:22.828131 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:22.810584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42847 (local address 127.0.97.62:46631)
0929 15:51:22.810995 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:22.810999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:22.811914 (+ 915us) server_negotiation.cc:407] Connection header received
0929 15:51:22.813778 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:22.813783 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:22.813871 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:22.813987 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:22.815425 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:22.816110 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:22.817813 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:22.818363 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:22.818498 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:22.822293 (+ 3795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:22.822298 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:22.822301 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:22.822333 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:22.825217 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:22.825944 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:22.825953 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:22.825956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:22.826064 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:22.826486 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:22.826491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:22.826495 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:22.826761 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:22.826919 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:22.827547 (+ 628us) server_negotiation.cc:299] Negotiation successful
0929 15:51:22.827761 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":227,"threads_started":1}
I20260929 15:51:22.829274 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:22.811095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46631 (local address 127.0.97.1:42847)
0929 15:51:22.811749 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:22.811785 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:22.813526 (+ 1741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:22.814213 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:22.814221 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:22.814709 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:51:22.815256 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:22.815272 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:22.816265 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:22.816273 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:22.817677 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:22.817689 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:22.818591 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:22.818598 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:22.818858 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:22.819664 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:51:22.819686 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:51:22.822061 (+ 2375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:51:22.825392 (+ 3331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:22.825400 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:22.825415 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:22.825816 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:22.826179 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:22.826184 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:22.826187 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:22.826370 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:22.826884 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:51:22.826892 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:51:22.827200 (+ 308us) client_negotiation.cc:769] Sending connection context
0929 15:51:22.827458 (+ 258us) client_negotiation.cc:240] Negotiation successful
0929 15:51:22.827953 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":262,"threads_started":1}
I20260929 15:51:22.831106 1151 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46631
I20260929 15:51:22.831419 1151 heartbeater.cc:464] Registering TS with master...
I20260929 15:51:22.832113 1151 heartbeater.cc:511] Master 127.0.97.62:46631 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 15:51:22.834246 962 ts_manager.cc:194] Registered new tserver with Master: 308c40d919db40f1a6de8dbd8a5ae524 (127.0.97.1:43241)
I20260929 15:51:22.835619 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42847
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 15:51:22.857697 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46631
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:51:23.170482 1157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:51:23.170744 1157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:51:23.170917 1157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:51:23.178115 1157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:51:23.178211 1157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:51:23.178395 1157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:51:23.188349 1157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46631
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:51:23.189710 1157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:51:23.191020 1157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:51:23.203732 1162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:23.203704 1165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:23.203768 1163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:51:23.204644 1157 server_base.cc:1034] running on GCE node
I20260929 15:51:23.205142 1157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:51:23.206143 1157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:51:23.207365 1157 hybrid_clock.cc:648] HybridClock initialized: now 1790697083207322 us; error 65 us; skew 500 ppm
I20260929 15:51:23.207674 1157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:51:23 dist-test-slave-r0ck krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:51:23.210716 1157 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:51:23.212110 1157 webserver.cc:460] Webserver started at http://127.0.97.2:40799/ using document root <none> and password file <none>
I20260929 15:51:23.212772 1157 fs_manager.cc:359] Metadata directory not provided
I20260929 15:51:23.212915 1157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:51:23.213215 1157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:51:23.216019 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-1/data/instance:
uuid: "d6507563a31e4dba8446cbc3d30e0ea2"
format_stamp: "Formatted at 2026-09-29 15:51:23 on dist-test-slave-r0ck"
server_key: "0a3ffa127382d84a1b91281bc4ac8bb7"
server_key_iv: "2ad71a451fe634f5d3963f661163edfd"
server_key_version: "encryptionkey@0"
I20260929 15:51:23.216820 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d6507563a31e4dba8446cbc3d30e0ea2"
format_stamp: "Formatted at 2026-09-29 15:51:23 on dist-test-slave-r0ck"
server_key: "0a3ffa127382d84a1b91281bc4ac8bb7"
server_key_iv: "2ad71a451fe634f5d3963f661163edfd"
server_key_version: "encryptionkey@0"
I20260929 15:51:23.221902 1157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 15:51:23.225263 1172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:23.226644 1157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:51:23.226773 1157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-1/wal
uuid: "d6507563a31e4dba8446cbc3d30e0ea2"
format_stamp: "Formatted at 2026-09-29 15:51:23 on dist-test-slave-r0ck"
server_key: "0a3ffa127382d84a1b91281bc4ac8bb7"
server_key_iv: "2ad71a451fe634f5d3963f661163edfd"
server_key_version: "encryptionkey@0"
I20260929 15:51:23.226948 1157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:51:23.242700 1157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:51:23.246914 1157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:51:23.247205 1157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:51:23.248097 1157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:51:23.249423 1157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:51:23.249522 1157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:23.249650 1157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:51:23.249720 1157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:23.285497 1157 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42947
I20260929 15:51:23.285517 1284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42947 every 8 connection(s)
I20260929 15:51:23.286840 1157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:51:23.293540 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 1157
I20260929 15:51:23.293751 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:51:23.294174 388 external_mini_cluster.cc:1428] Setting key 2015d03859a8f26031bb0231ee86a19d
Sep 29 15:51:23 dist-test-slave-r0ck krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:51:23.307936 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:23.290014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44353 (local address 127.0.97.62:46631)
0929 15:51:23.290289 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:23.290318 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:23.291622 (+ 1304us) server_negotiation.cc:407] Connection header received
0929 15:51:23.293418 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:23.293432 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:23.293516 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:23.293634 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:23.295386 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:23.296295 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:23.297989 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:23.298751 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:23.298897 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:23.302321 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:23.302327 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:23.302332 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:23.302376 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:23.304841 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:23.305687 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:23.305694 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:23.305699 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:23.305860 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:23.306323 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:23.306328 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:23.306331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:23.306605 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:23.306756 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:23.307405 (+ 649us) server_negotiation.cc:299] Negotiation successful
0929 15:51:23.307615 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260929 15:51:23.308895 1287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:23.290497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46631 (local address 127.0.97.2:44353)
0929 15:51:23.291471 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:23.291511 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:23.293138 (+ 1627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:23.293894 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:23.293905 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:23.294464 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:51:23.295193 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:23.295220 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:23.296481 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:23.296485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:23.297832 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:23.297844 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:23.298976 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:23.298981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:23.299174 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:23.299860 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:51:23.299882 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:51:23.302122 (+ 2240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:51:23.304975 (+ 2853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:23.304983 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:23.304999 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:23.305540 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:23.305997 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:23.306002 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:23.306006 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:23.306189 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:23.306728 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:51:23.306736 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:51:23.307046 (+ 310us) client_negotiation.cc:769] Sending connection context
0929 15:51:23.307298 (+ 252us) client_negotiation.cc:240] Negotiation successful
0929 15:51:23.307774 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":399,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 15:51:23.310328 1285 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46631
I20260929 15:51:23.310642 1285 heartbeater.cc:464] Registering TS with master...
I20260929 15:51:23.311354 1285 heartbeater.cc:511] Master 127.0.97.62:46631 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 15:51:23.312763 962 ts_manager.cc:194] Registered new tserver with Master: d6507563a31e4dba8446cbc3d30e0ea2 (127.0.97.2:42947)
I20260929 15:51:23.313638 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:44353
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 15:51:23.333248 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46631
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:51:23.644523 1290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:51:23.644781 1290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:51:23.644948 1290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:51:23.651960 1290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:51:23.652060 1290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:51:23.652233 1290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:51:23.661470 1290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46631
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:51:23.662693 1290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:51:23.664003 1290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:51:23.676416 1295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:23.676553 1298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:23.676518 1296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:51:23.677390 1290 server_base.cc:1034] running on GCE node
I20260929 15:51:23.677896 1290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:51:23.678745 1290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:51:23.679944 1290 hybrid_clock.cc:648] HybridClock initialized: now 1790697083679910 us; error 30 us; skew 500 ppm
I20260929 15:51:23.680204 1290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:51:23 dist-test-slave-r0ck krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:51:23.683316 1290 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:51:23.684556 1290 webserver.cc:460] Webserver started at http://127.0.97.3:44929/ using document root <none> and password file <none>
I20260929 15:51:23.685148 1290 fs_manager.cc:359] Metadata directory not provided
I20260929 15:51:23.685238 1290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:51:23.685561 1290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:51:23.688383 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-2/data/instance:
uuid: "448bf0ab0f914619ba6aba55613fe96d"
format_stamp: "Formatted at 2026-09-29 15:51:23 on dist-test-slave-r0ck"
server_key: "fe161fc0bab6025d3367b353ff84bac4"
server_key_iv: "dc57e74303211be78385c7379ce87855"
server_key_version: "encryptionkey@0"
I20260929 15:51:23.689147 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "448bf0ab0f914619ba6aba55613fe96d"
format_stamp: "Formatted at 2026-09-29 15:51:23 on dist-test-slave-r0ck"
server_key: "fe161fc0bab6025d3367b353ff84bac4"
server_key_iv: "dc57e74303211be78385c7379ce87855"
server_key_version: "encryptionkey@0"
I20260929 15:51:23.694150 1290 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260929 15:51:23.697460 1305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:23.698714 1290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 15:51:23.698900 1290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-2/wal
uuid: "448bf0ab0f914619ba6aba55613fe96d"
format_stamp: "Formatted at 2026-09-29 15:51:23 on dist-test-slave-r0ck"
server_key: "fe161fc0bab6025d3367b353ff84bac4"
server_key_iv: "dc57e74303211be78385c7379ce87855"
server_key_version: "encryptionkey@0"
I20260929 15:51:23.699093 1290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:51:23.714983 1290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:51:23.719158 1290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:51:23.719436 1290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:51:23.720312 1290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:51:23.721606 1290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:51:23.721673 1290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:23.721825 1290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:51:23.721876 1290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:23.752161 1290 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35543
I20260929 15:51:23.752199 1417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35543 every 8 connection(s)
I20260929 15:51:23.753398 1290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:51:23.757869 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 1290
I20260929 15:51:23.758061 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697075657932-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:51:23.758467 388 external_mini_cluster.cc:1428] Setting key d43c35ea909c2877194d9979d5ae90ee
Sep 29 15:51:23 dist-test-slave-r0ck krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:51:23.771883 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:23.756209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40569 (local address 127.0.97.62:46631)
0929 15:51:23.756380 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:23.756385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:23.757537 (+ 1152us) server_negotiation.cc:407] Connection header received
0929 15:51:23.759488 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:23.759492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:23.759577 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:23.759679 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:23.761076 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:23.761937 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:23.763383 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:23.763780 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:23.763881 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:23.767024 (+ 3143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:23.767033 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:23.767036 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:23.767069 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:23.769393 (+ 2324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:23.770003 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:23.770007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:23.770009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:23.770116 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:23.770444 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:23.770447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:23.770449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:23.770687 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:23.770787 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:23.771370 (+ 583us) server_negotiation.cc:299] Negotiation successful
0929 15:51:23.771567 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 15:51:23.772881 1420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:23.756684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46631 (local address 127.0.97.3:40569)
0929 15:51:23.757331 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:23.757430 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:23.759212 (+ 1782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:23.759895 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:23.759909 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:23.760369 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:51:23.760880 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:23.760928 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:23.762067 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:23.762071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:23.763257 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:23.763268 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:23.763962 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:23.763966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:23.764144 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:23.764730 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:51:23.764750 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:51:23.766834 (+ 2084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:51:23.769519 (+ 2685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:23.769524 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:23.769536 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:23.769911 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:23.770200 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:23.770203 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:23.770205 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:23.770358 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:23.770780 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:51:23.770785 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:51:23.771064 (+ 279us) client_negotiation.cc:769] Sending connection context
0929 15:51:23.771265 (+ 201us) client_negotiation.cc:240] Negotiation successful
0929 15:51:23.771600 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":262,"threads_started":1}
I20260929 15:51:23.774217 1418 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46631
I20260929 15:51:23.774500 1418 heartbeater.cc:464] Registering TS with master...
I20260929 15:51:23.775566 1418 heartbeater.cc:511] Master 127.0.97.62:46631 requested a full tablet report, sending...
I20260929 15:51:23.776964 962 ts_manager.cc:194] Registered new tserver with Master: 448bf0ab0f914619ba6aba55613fe96d (127.0.97.3:35543)
I20260929 15:51:23.777789 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40569
I20260929 15:51:23.785602 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:51:23.805027 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:23.791745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33638 (local address 127.0.97.62:46631)
0929 15:51:23.791940 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:23.791947 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:23.792467 (+ 520us) server_negotiation.cc:407] Connection header received
0929 15:51:23.792595 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:23.792600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:23.792659 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:23.792740 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:23.794277 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:23.794964 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:23.796186 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:23.796588 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:23.796678 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:23.798877 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:23.798883 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:23.798887 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:23.798919 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:23.801504 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:23.802378 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:23.802386 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:23.802390 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:23.802523 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:23.802993 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:23.802998 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:23.803001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:23.803317 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:23.803593 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:23.804422 (+ 829us) server_negotiation.cc:299] Negotiation successful
0929 15:51:23.804657 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 15:51:23.812192 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:51:23.819034 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:51:23.839190 1151 heartbeater.cc:503] Master 127.0.97.62:46631 was elected leader, sending a full tablet report...
I20260929 15:51:23.853711 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:23.830536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43241 (local address 127.0.0.1:47852)
0929 15:51:23.840554 (+ 10018us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:23.840588 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:23.840751 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:23.847056 (+ 6305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:23.847063 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:23.847094 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:51:23.847571 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:23.847583 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:23.849413 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:23.849421 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:23.851169 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:23.851180 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:23.852436 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:23.852442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:23.852971 (+ 529us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:23.853025 (+ 54us) client_negotiation.cc:769] Sending connection context
0929 15:51:23.853162 (+ 137us) client_negotiation.cc:240] Negotiation successful
0929 15:51:23.853373 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9790,"thread_start_us":15722,"threads_started":1}
I20260929 15:51:23.853706 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:23.830728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47852 (local address 127.0.97.1:43241)
0929 15:51:23.846053 (+ 15325us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:23.846062 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:23.846095 (+ 33us) server_negotiation.cc:407] Connection header received
0929 15:51:23.846199 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:23.846207 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:23.846491 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:23.847768 (+ 1277us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:51:23.848007 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:23.849232 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:23.851327 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:23.852236 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:23.852404 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:23.852637 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:23.853199 (+ 562us) server_negotiation.cc:299] Negotiation successful
0929 15:51:23.853392 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15157,"thread_start_us":15893,"threads_started":1}
I20260929 15:51:23.859198 1086 tablet_service.cc:1467] Processing CreateTablet for tablet 2d84538085c642279e0519f2ed06bfd9 (DEFAULT_TABLE table=test-table [id=a287e71a240746878eaa9f2e0e48effb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:51:23.860635 1086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d84538085c642279e0519f2ed06bfd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:23.862452 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:23.831247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35543 (local address 127.0.0.1:47204)
0929 15:51:23.853961 (+ 22714us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:23.853981 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:23.854070 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:23.855724 (+ 1654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:23.855732 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:23.855770 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:51:23.856038 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:23.856052 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:23.858311 (+ 2259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:23.858318 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:23.860016 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:23.860029 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:23.861702 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:23.861708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:23.861896 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:23.861940 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 15:51:23.862039 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 15:51:23.862175 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22585,"thread_start_us":8959,"threads_started":1}
I20260929 15:51:23.862566 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:23.832019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47204 (local address 127.0.97.3:35543)
0929 15:51:23.855172 (+ 23153us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:23.855179 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:23.855208 (+ 29us) server_negotiation.cc:407] Connection header received
0929 15:51:23.855294 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:23.855301 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:23.855506 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:23.855696 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:51:23.857135 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:23.858162 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:23.860181 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:23.861540 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:23.861675 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:23.861897 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:23.862100 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 15:51:23.862257 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22991,"thread_start_us":22841,"threads_started":1}
I20260929 15:51:23.864028 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:23.830875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42947 (local address 127.0.0.1:33646)
0929 15:51:23.846057 (+ 15182us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:23.846077 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:23.846192 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:23.855966 (+ 9774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:23.855970 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:23.855999 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:51:23.856938 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:23.856950 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:23.859427 (+ 2477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:23.859432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:23.861028 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:23.861039 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:23.863262 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:23.863268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:23.863455 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:23.863484 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 15:51:23.863569 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 15:51:23.863698 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15071,"thread_start_us":25843,"threads_started":1}
I20260929 15:51:23.864429 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:23.832510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33646 (local address 127.0.97.2:42947)
0929 15:51:23.853078 (+ 20568us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:23.853086 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:23.853114 (+ 28us) server_negotiation.cc:407] Connection header received
0929 15:51:23.854057 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:23.854064 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:23.854313 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:23.854529 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:51:23.857160 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:23.859282 (+ 2122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:23.861167 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:23.862468 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:23.862935 (+ 467us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:23.863126 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:23.863941 (+ 815us) server_negotiation.cc:299] Negotiation successful
0929 15:51:23.864117 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20422,"thread_start_us":20280,"threads_started":1}
I20260929 15:51:23.866127 1353 tablet_service.cc:1467] Processing CreateTablet for tablet 2d84538085c642279e0519f2ed06bfd9 (DEFAULT_TABLE table=test-table [id=a287e71a240746878eaa9f2e0e48effb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:51:23.867534 1353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d84538085c642279e0519f2ed06bfd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:23.867468 1220 tablet_service.cc:1467] Processing CreateTablet for tablet 2d84538085c642279e0519f2ed06bfd9 (DEFAULT_TABLE table=test-table [id=a287e71a240746878eaa9f2e0e48effb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:51:23.868502 1220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d84538085c642279e0519f2ed06bfd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:23.875190 1436 tablet_bootstrap.cc:492] T 2d84538085c642279e0519f2ed06bfd9 P 308c40d919db40f1a6de8dbd8a5ae524: Bootstrap starting.
I20260929 15:51:23.879783 1436 tablet_bootstrap.cc:654] T 2d84538085c642279e0519f2ed06bfd9 P 308c40d919db40f1a6de8dbd8a5ae524: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:23.881129 1436 log.cc:824] T 2d84538085c642279e0519f2ed06bfd9 P 308c40d919db40f1a6de8dbd8a5ae524: Log is configured to *not* fsync() on all Append() calls
I20260929 15:51:23.881501 1437 tablet_bootstrap.cc:492] T 2d84538085c642279e0519f2ed06bfd9 P d6507563a31e4dba8446cbc3d30e0ea2: Bootstrap starting.
I20260929 15:51:23.884183 1439 tablet_bootstrap.cc:492] T 2d84538085c642279e0519f2ed06bfd9 P 448bf0ab0f914619ba6aba55613fe96d: Bootstrap starting.
I20260929 15:51:23.885844 1437 tablet_bootstrap.cc:654] T 2d84538085c642279e0519f2ed06bfd9 P d6507563a31e4dba8446cbc3d30e0ea2: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:23.887151 1437 log.cc:824] T 2d84538085c642279e0519f2ed06bfd9 P d6507563a31e4dba8446cbc3d30e0ea2: Log is configured to *not* fsync() on all Append() calls
I20260929 15:51:23.887418 1439 tablet_bootstrap.cc:654] T 2d84538085c642279e0519f2ed06bfd9 P 448bf0ab0f914619ba6aba55613fe96d: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:23.887486 1436 tablet_bootstrap.cc:492] T 2d84538085c642279e0519f2ed06bfd9 P 308c40d919db40f1a6de8dbd8a5ae524: No bootstrap required, opened a new log
I20260929 15:51:23.887759 1436 ts_tablet_manager.cc:1397] T 2d84538085c642279e0519f2ed06bfd9 P 308c40d919db40f1a6de8dbd8a5ae524: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.006s
I20260929 15:51:23.888571 1439 log.cc:824] T 2d84538085c642279e0519f2ed06bfd9 P 448bf0ab0f914619ba6aba55613fe96d: Log is configured to *not* fsync() on all Append() calls
I20260929 15:51:23.891868 1439 tablet_bootstrap.cc:492] T 2d84538085c642279e0519f2ed06bfd9 P 448bf0ab0f914619ba6aba55613fe96d: No bootstrap required, opened a new log
I20260929 15:51:23.892115 1439 ts_tablet_manager.cc:1397] T 2d84538085c642279e0519f2ed06bfd9 P 448bf0ab0f914619ba6aba55613fe96d: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260929 15:51:23.893538 1437 tablet_bootstrap.cc:492] T 2d84538085c642279e0519f2ed06bfd9 P d6507563a31e4dba8446cbc3d30e0ea2: No bootstrap required, opened a new log
I20260929 15:51:23.893774 1437 ts_tablet_manager.cc:1397] T 2d84538085c642279e0519f2ed06bfd9 P d6507563a31e4dba8446cbc3d30e0ea2: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20260929 15:51:23.898736 1436 raft_consensus.cc:348] T 2d84538085c642279e0519f2ed06bfd9 P 308c40d919db40f1a6de8dbd8a5ae524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "308c40d919db40f1a6de8dbd8a5ae524" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43241 } } peers { permanent_uuid: "d6507563a31e4dba8446cbc3d30e0ea2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42947 } } peers { permanent_uuid: "448bf0ab0f914619ba6aba55613fe96d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 15:51:23.899173 1436 raft_consensus.cc:374] T 2d84538085c642279e0519f2ed06bfd9 P 308c40d919db40f1a6de8dbd8a5ae524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:23.899263 1436 raft_consensus.cc:729] T 2d84538085c642279e0519f2ed06bfd9 P 308c40d919db40f1a6de8dbd8a5ae524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 308c40d919db40f1a6de8dbd8a5ae524, State: Initialized, Role: FOLLOWER
I20260929 15:51:23.899850 1436 consensus_queue.cc:260] T 2d84538085c642279e0519f2ed06bfd9 P 308c40d919db40f1a6de8dbd8a5ae524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "308c40d919db40f1a6de8dbd8a5ae524" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43241 } } peers { permanent_uuid: "d6507563a31e4dba8446cbc3d30e0ea2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42947 } } peers { permanent_uuid: "448bf0ab0f914619ba6aba55613fe96d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 15:51:23.901619 1436 ts_tablet_manager.cc:1428] T 2d84538085c642279e0519f2ed06bfd9 P 308c40d919db40f1a6de8dbd8a5ae524: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260929 15:51:23.901729 1437 raft_consensus.cc:348] T 2d84538085c642279e0519f2ed06bfd9 P d6507563a31e4dba8446cbc3d30e0ea2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "308c40d919db40f1a6de8dbd8a5ae524" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43241 } } peers { permanent_uuid: "d6507563a31e4dba8446cbc3d30e0ea2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42947 } } peers { permanent_uuid: "448bf0ab0f914619ba6aba55613fe96d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 15:51:23.902156 1437 raft_consensus.cc:374] T 2d84538085c642279e0519f2ed06bfd9 P d6507563a31e4dba8446cbc3d30e0ea2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:23.902266 1437 raft_consensus.cc:729] T 2d84538085c642279e0519f2ed06bfd9 P d6507563a31e4dba8446cbc3d30e0ea2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6507563a31e4dba8446cbc3d30e0ea2, State: Initialized, Role: FOLLOWER
I20260929 15:51:23.902333 1439 raft_consensus.cc:348] T 2d84538085c642279e0519f2ed06bfd9 P 448bf0ab0f914619ba6aba55613fe96d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "308c40d919db40f1a6de8dbd8a5ae524" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43241 } } peers { permanent_uuid: "d6507563a31e4dba8446cbc3d30e0ea2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42947 } } peers { permanent_uuid: "448bf0ab0f914619ba6aba55613fe96d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 15:51:23.902779 1439 raft_consensus.cc:374] T 2d84538085c642279e0519f2ed06bfd9 P 448bf0ab0f914619ba6aba55613fe96d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:23.902869 1439 raft_consensus.cc:729] T 2d84538085c642279e0519f2ed06bfd9 P 448bf0ab0f914619ba6aba55613fe96d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 448bf0ab0f914619ba6aba55613fe96d, State: Initialized, Role: FOLLOWER
I20260929 15:51:23.902839 1437 consensus_queue.cc:260] T 2d84538085c642279e0519f2ed06bfd9 P d6507563a31e4dba8446cbc3d30e0ea2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "308c40d919db40f1a6de8dbd8a5ae524" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43241 } } peers { permanent_uuid: "d6507563a31e4dba8446cbc3d30e0ea2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42947 } } peers { permanent_uuid: "448bf0ab0f914619ba6aba55613fe96d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 15:51:23.903357 1439 consensus_queue.cc:260] T 2d84538085c642279e0519f2ed06bfd9 P 448bf0ab0f914619ba6aba55613fe96d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "308c40d919db40f1a6de8dbd8a5ae524" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43241 } } peers { permanent_uuid: "d6507563a31e4dba8446cbc3d30e0ea2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42947 } } peers { permanent_uuid: "448bf0ab0f914619ba6aba55613fe96d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 15:51:23.904414 1437 ts_tablet_manager.cc:1428] T 2d84538085c642279e0519f2ed06bfd9 P d6507563a31e4dba8446cbc3d30e0ea2: Time spent starting tablet: real 0.011s user 0.009s sys 0.001s
I20260929 15:51:23.904448 1418 heartbeater.cc:503] Master 127.0.97.62:46631 was elected leader, sending a full tablet report...
I20260929 15:51:23.904783 1285 heartbeater.cc:503] Master 127.0.97.62:46631 was elected leader, sending a full tablet report...
I20260929 15:51:23.905035 1439 ts_tablet_manager.cc:1428] T 2d84538085c642279e0519f2ed06bfd9 P 448bf0ab0f914619ba6aba55613fe96d: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
W20260929 15:51:24.005457 1419 tablet.cc:2369] T 2d84538085c642279e0519f2ed06bfd9 P 448bf0ab0f914619ba6aba55613fe96d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:51:24.026609 1443 raft_consensus.cc:482] T 2d84538085c642279e0519f2ed06bfd9 P d6507563a31e4dba8446cbc3d30e0ea2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:51:24.026877 1443 raft_consensus.cc:504] T 2d84538085c642279e0519f2ed06bfd9 P d6507563a31e4dba8446cbc3d30e0ea2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "308c40d919db40f1a6de8dbd8a5ae524" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43241 } } peers { permanent_uuid: "d6507563a31e4dba8446cbc3d30e0ea2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42947 } } peers { permanent_uuid: "448bf0ab0f914619ba6aba55613fe96d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 15:51:24.028564 1443 leader_election.cc:288] T 2d84538085c642279e0519f2ed06bfd9 P d6507563a31e4dba8446cbc3d30e0ea2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 308c40d919db40f1a6de8dbd8a5ae524 (127.0.97.1:43241), 448bf0ab0f914619ba6aba55613fe96d (127.0.97.3:35543)
I20260929 15:51:24.038035 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:24.028935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39397 (local address 127.0.97.1:43241)
0929 15:51:24.029167 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:24.029176 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:24.029951 (+ 775us) server_negotiation.cc:407] Connection header received
0929 15:51:24.030213 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:24.030217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:24.030275 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:24.030334 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:51:24.031165 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:24.032219 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:24.035309 (+ 3090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:24.036499 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:24.036647 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:24.036746 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:24.037631 (+ 885us) server_negotiation.cc:299] Negotiation successful
0929 15:51:24.037763 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 15:51:24.038067 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:24.029202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60605 (local address 127.0.97.3:35543)
0929 15:51:24.029448 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:24.029455 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:24.030010 (+ 555us) server_negotiation.cc:407] Connection header received
0929 15:51:24.030219 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:24.030224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:24.030290 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:24.030381 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:51:24.031015 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:24.032164 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:24.035371 (+ 3207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:24.036540 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:24.036661 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:24.036747 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:24.037633 (+ 886us) server_negotiation.cc:299] Negotiation successful
0929 15:51:24.037762 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 15:51:24.038267 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:24.029002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35543 (local address 127.0.97.2:60605)
0929 15:51:24.029902 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:24.029916 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:24.030075 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:24.030408 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:24.030413 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:24.030437 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:51:24.030858 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:24.030871 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:24.032364 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:24.032370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:24.035142 (+ 2772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:24.035150 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:24.037234 (+ 2084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:24.037242 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:24.037454 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:24.037489 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 15:51:24.037611 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 15:51:24.037768 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":809,"mutex_wait_us":226,"spinlock_wait_cycles":22400,"thread_start_us":613,"threads_started":1}
I20260929 15:51:24.038291 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:24.028818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43241 (local address 127.0.97.2:39397)
0929 15:51:24.029825 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:24.029846 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:24.030030 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:24.030498 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:24.030501 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:24.030529 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:51:24.031032 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:24.031043 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:24.032366 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:24.032370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:24.035131 (+ 2761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:24.035146 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:24.037266 (+ 2120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:24.037271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:24.037454 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:24.037478 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 15:51:24.037606 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 15:51:24.037762 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":806,"mutex_wait_us":181,"thread_start_us":548,"threads_started":1}
W20260929 15:51:24.038772 1286 tablet.cc:2369] T 2d84538085c642279e0519f2ed06bfd9 P d6507563a31e4dba8446cbc3d30e0ea2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:51:24.039265 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d84538085c642279e0519f2ed06bfd9" candidate_uuid: "d6507563a31e4dba8446cbc3d30e0ea2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "308c40d919db40f1a6de8dbd8a5ae524" is_pre_election: true
I20260929 15:51:24.039743 1106 raft_consensus.cc:2454] T 2d84538085c642279e0519f2ed06bfd9 P 308c40d919db40f1a6de8dbd8a5ae524 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6507563a31e4dba8446cbc3d30e0ea2 in term 0.
I20260929 15:51:24.040395 1174 leader_election.cc:302] T 2d84538085c642279e0519f2ed06bfd9 P d6507563a31e4dba8446cbc3d30e0ea2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 308c40d919db40f1a6de8dbd8a5ae524, d6507563a31e4dba8446cbc3d30e0ea2; no voters:
I20260929 15:51:24.040534 1373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d84538085c642279e0519f2ed06bfd9" candidate_uuid: "d6507563a31e4dba8446cbc3d30e0ea2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "448bf0ab0f914619ba6aba55613fe96d" is_pre_election: true
I20260929 15:51:24.040835 1443 raft_consensus.cc:2781] T 2d84538085c642279e0519f2ed06bfd9 P d6507563a31e4dba8446cbc3d30e0ea2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:51:24.041013 1443 raft_consensus.cc:482] T 2d84538085c642279e0519f2ed06bfd9 P d6507563a31e4dba8446cbc3d30e0ea2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:51:24.040993 1373 raft_consensus.cc:2454] T 2d84538085c642279e0519f2ed06bfd9 P 448bf0ab0f914619ba6aba55613fe96d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6507563a31e4dba8446cbc3d30e0ea2 in term 0.
I20260929 15:51:24.041133 1443 raft_consensus.cc:3037] T 2d84538085c642279e0519f2ed06bfd9 P d6507563a31e4dba8446cbc3d30e0ea2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:24.043514 1443 raft_consensus.cc:504] T 2d84538085c642279e0519f2ed06bfd9 P d6507563a31e4dba8446cbc3d30e0ea2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "308c40d919db40f1a6de8dbd8a5ae524" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43241 } } peers { permanent_uuid: "d6507563a31e4dba8446cbc3d30e0ea2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42947 } } peers { permanent_uuid: "448bf0ab0f914619ba6aba55613fe96d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 15:51:24.044145 1443 leader_election.cc:288] T 2d84538085c642279e0519f2ed06bfd9 P d6507563a31e4dba8446cbc3d30e0ea2 [CANDIDATE]: Term 1 election: Requested vote from peers 308c40d919db40f1a6de8dbd8a5ae524 (127.0.97.1:43241), 448bf0ab0f914619ba6aba55613fe96d (127.0.97.3:35543)
I20260929 15:51:24.044615 1373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d84538085c642279e0519f2ed06bfd9" candidate_uuid: "d6507563a31e4dba8446cbc3d30e0ea2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "448bf0ab0f914619ba6aba55613fe96d"
I20260929 15:51:24.044665 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d84538085c642279e0519f2ed06bfd9" candidate_uuid: "d6507563a31e4dba8446cbc3d30e0ea2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "308c40d919db40f1a6de8dbd8a5ae524"
I20260929 15:51:24.044824 1373 raft_consensus.cc:3037] T 2d84538085c642279e0519f2ed06bfd9 P 448bf0ab0f914619ba6aba55613fe96d [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:24.044847 1106 raft_consensus.cc:3037] T 2d84538085c642279e0519f2ed06bfd9 P 308c40d919db40f1a6de8dbd8a5ae524 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:24.047986 1106 raft_consensus.cc:2454] T 2d84538085c642279e0519f2ed06bfd9 P 308c40d919db40f1a6de8dbd8a5ae524 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6507563a31e4dba8446cbc3d30e0ea2 in term 1.
I20260929 15:51:24.047986 1373 raft_consensus.cc:2454] T 2d84538085c642279e0519f2ed06bfd9 P 448bf0ab0f914619ba6aba55613fe96d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6507563a31e4dba8446cbc3d30e0ea2 in term 1.
I20260929 15:51:24.048409 1173 leader_election.cc:302] T 2d84538085c642279e0519f2ed06bfd9 P d6507563a31e4dba8446cbc3d30e0ea2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 448bf0ab0f914619ba6aba55613fe96d, d6507563a31e4dba8446cbc3d30e0ea2; no voters:
I20260929 15:51:24.048717 1443 raft_consensus.cc:2781] T 2d84538085c642279e0519f2ed06bfd9 P d6507563a31e4dba8446cbc3d30e0ea2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:51:24.049202 1443 raft_consensus.cc:686] T 2d84538085c642279e0519f2ed06bfd9 P d6507563a31e4dba8446cbc3d30e0ea2 [term 1 LEADER]: Becoming Leader. State: Replica: d6507563a31e4dba8446cbc3d30e0ea2, State: Running, Role: LEADER
I20260929 15:51:24.049695 1443 consensus_queue.cc:237] T 2d84538085c642279e0519f2ed06bfd9 P d6507563a31e4dba8446cbc3d30e0ea2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "308c40d919db40f1a6de8dbd8a5ae524" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43241 } } peers { permanent_uuid: "d6507563a31e4dba8446cbc3d30e0ea2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42947 } } peers { permanent_uuid: "448bf0ab0f914619ba6aba55613fe96d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 15:51:24.054656 960 catalog_manager.cc:5495] T 2d84538085c642279e0519f2ed06bfd9 P d6507563a31e4dba8446cbc3d30e0ea2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6507563a31e4dba8446cbc3d30e0ea2 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "d6507563a31e4dba8446cbc3d30e0ea2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "308c40d919db40f1a6de8dbd8a5ae524" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6507563a31e4dba8446cbc3d30e0ea2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "448bf0ab0f914619ba6aba55613fe96d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } health_report { overall_health: UNKNOWN } } }
W20260929 15:51:24.060565 1152 tablet.cc:2369] T 2d84538085c642279e0519f2ed06bfd9 P 308c40d919db40f1a6de8dbd8a5ae524: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 15:51:24 dist-test-slave-r0ck krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 15:51:24.100536 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:24.085509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47854 (local address 127.0.97.1:43241)
0929 15:51:24.085745 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:24.085755 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:24.086098 (+ 343us) server_negotiation.cc:407] Connection header received
0929 15:51:24.086232 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:24.086236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:24.086313 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:24.086396 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:24.087931 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:24.088887 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:24.090299 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:24.090729 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:24.090836 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:24.095302 (+ 4466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:24.095318 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:24.095334 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:24.095370 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:24.098106 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:24.098733 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:24.098741 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:24.098743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:24.098874 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:24.099189 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:24.099192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:24.099194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:24.099540 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:24.099758 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:24.099983 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 15:51:24.100165 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Sep 29 15:51:24 dist-test-slave-r0ck krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697084, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 15:51:24 dist-test-slave-r0ck krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697084, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 15:51:24.144039 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:24.130915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47870 (local address 127.0.97.1:43241)
0929 15:51:24.131160 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:24.131168 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:24.131504 (+ 336us) server_negotiation.cc:407] Connection header received
0929 15:51:24.131710 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:24.131714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:24.131780 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:24.131872 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:24.133088 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:24.133919 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:24.135122 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:24.135515 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:24.135608 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:24.139239 (+ 3631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:24.139243 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:24.139246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:24.139276 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:24.141767 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:24.142341 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:24.142345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:24.142348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:24.142460 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:24.142807 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:24.142810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:24.142812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:24.143049 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:24.143299 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:24.143542 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 15:51:24.143695 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
Sep 29 15:51:24 dist-test-slave-r0ck krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 15:51:24 dist-test-slave-r0ck krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 15:51:24.185451 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:24.172429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47886 (local address 127.0.97.1:43241)
0929 15:51:24.172670 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:24.172677 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:24.172943 (+ 266us) server_negotiation.cc:407] Connection header received
0929 15:51:24.173131 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:24.173136 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:24.173219 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:24.173311 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:24.174726 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:24.175397 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:24.176573 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:24.177006 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:24.177099 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:24.180820 (+ 3721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:24.180825 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:24.180828 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:24.180868 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:24.183336 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:24.183904 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:24.183908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:24.183910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:24.184013 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:24.184291 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:24.184294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:24.184295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:24.184564 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:24.184737 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:24.184922 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 15:51:24.185080 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
W20260929 15:51:24.187107 1066 tablet_service.cc:2908] Rejecting scan request for tablet 2d84538085c642279e0519f2ed06bfd9: Uninitialized: clean time has not yet been initialized
I20260929 15:51:24.190564 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 1023
I20260929 15:51:24.202231 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:24.191292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47894 (local address 127.0.97.1:43241)
0929 15:51:24.191501 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:24.191507 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:24.191711 (+ 204us) server_negotiation.cc:407] Connection header received
0929 15:51:24.191807 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:24.191811 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:24.191876 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:24.191959 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:24.193000 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:24.193704 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:24.195233 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:24.195613 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:24.195710 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:24.197741 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:24.197745 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:24.197748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:24.197774 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:24.200143 (+ 2369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:24.200690 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:24.200694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:24.200696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:24.200801 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:24.201082 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:24.201085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:24.201087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:24.201328 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:24.201597 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:24.201807 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 15:51:24.201955 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 15:51:24.202993 1146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:51:24.301671 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 1023
I20260929 15:51:24.319440 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 1157
Sep 29 15:51:24 dist-test-slave-r0ck krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 15:51:24.333947 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:24.320173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33654 (local address 127.0.97.2:42947)
0929 15:51:24.320354 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:24.320360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:24.320409 (+ 49us) server_negotiation.cc:407] Connection header received
0929 15:51:24.320627 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:24.320632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:24.320699 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:24.320801 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:24.322002 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:24.322717 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:24.323983 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:24.324363 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:24.324495 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:24.328350 (+ 3855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:24.328361 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:24.328375 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:24.328405 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:24.331184 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:24.331818 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:24.331826 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:24.331828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:24.331982 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:24.332340 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:24.332347 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:24.332350 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:24.332648 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:24.332953 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:24.333175 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 15:51:24.333693 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 15:51:24.334573 1280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:51:24.422827 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 1157
I20260929 15:51:24.438892 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 1290
Sep 29 15:51:24 dist-test-slave-r0ck krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 15:51:24.453137 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:24.439597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47218 (local address 127.0.97.3:35543)
0929 15:51:24.439786 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:24.439792 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:24.439812 (+ 20us) server_negotiation.cc:407] Connection header received
0929 15:51:24.440000 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:24.440013 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:24.440075 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:24.440182 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:24.441383 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:24.442094 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:24.443323 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:24.443695 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:24.443826 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:24.447797 (+ 3971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:24.447809 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:24.447825 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:24.447867 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:24.450533 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:24.451162 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:24.451169 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:24.451172 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:24.451301 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:24.451667 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:24.451673 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:24.451676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:24.452011 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:24.452288 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:24.452600 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 15:51:24.452821 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 15:51:24.453852 1413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:51:24.538432 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 1290
I20260929 15:51:24.554338 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 930
I20260929 15:51:24.555500 992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:51:24.661059 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 930
2026-09-29T15:51:24Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4085 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:51:24 dist-test-slave-r0ck krb5kdc[1466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:51:24 dist-test-slave-r0ck krb5kdc[1466](info): set up 2 sockets
Sep 29 15:51:24 dist-test-slave-r0ck krb5kdc[1466](info): commencing operation
krb5kdc: starting...
W20260929 15:51:25.737241 388 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.003s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:51:25 dist-test-slave-r0ck krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:51:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:51:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:51:25.918051 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33219
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:51:26.226893 1479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:51:26.227169 1479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:51:26.227330 1479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:51:26.234396 1479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:51:26.234494 1479 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 15:51:26.234570 1479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:51:26.234658 1479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:51:26.234769 1479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:51:26.234875 1479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:51:26.244379 1479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33219
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:51:26.245724 1479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:51:26.246991 1479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:51:26.258639 1487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:26.258694 1484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:26.258724 1485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:51:26.259415 1479 server_base.cc:1034] running on GCE node
I20260929 15:51:26.260182 1479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:51:26.261548 1479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:51:26.262761 1479 hybrid_clock.cc:648] HybridClock initialized: now 1790697086262685 us; error 49 us; skew 500 ppm
I20260929 15:51:26.263091 1479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:51:26 dist-test-slave-r0ck krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:51:26.266139 1479 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:51:26.267498 1479 webserver.cc:460] Webserver started at http://127.0.97.62:46047/ using document root <none> and password file <none>
I20260929 15:51:26.268177 1479 fs_manager.cc:359] Metadata directory not provided
I20260929 15:51:26.268311 1479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:51:26.268635 1479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:51:26.271590 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "e6f7e0dcaa3c4ecd9658ee15fb3594ad"
format_stamp: "Formatted at 2026-09-29 15:51:26 on dist-test-slave-r0ck"
server_key: "35cd32626df5003ba40da0f40449a4ea"
server_key_iv: "0d0c5a1cf67ddaadacafe5fc605014e3"
server_key_version: "encryptionkey@0"
I20260929 15:51:26.272439 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "e6f7e0dcaa3c4ecd9658ee15fb3594ad"
format_stamp: "Formatted at 2026-09-29 15:51:26 on dist-test-slave-r0ck"
server_key: "35cd32626df5003ba40da0f40449a4ea"
server_key_iv: "0d0c5a1cf67ddaadacafe5fc605014e3"
server_key_version: "encryptionkey@0"
I20260929 15:51:26.277479 1479 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260929 15:51:26.280884 1494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:26.282269 1479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:51:26.282473 1479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "e6f7e0dcaa3c4ecd9658ee15fb3594ad"
format_stamp: "Formatted at 2026-09-29 15:51:26 on dist-test-slave-r0ck"
server_key: "35cd32626df5003ba40da0f40449a4ea"
server_key_iv: "0d0c5a1cf67ddaadacafe5fc605014e3"
server_key_version: "encryptionkey@0"
I20260929 15:51:26.282693 1479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:51:26.302069 1479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:51:26.306011 1479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:51:26.306267 1479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:51:26.326926 1479 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33219
I20260929 15:51:26.326965 1555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33219 every 8 connection(s)
I20260929 15:51:26.328189 1479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:51:26.331725 1556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:26.333691 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 1479
I20260929 15:51:26.333901 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:51:26.334527 388 external_mini_cluster.cc:1428] Setting key 1fe7184847df2a118e278ade2e638ec0
I20260929 15:51:26.344341 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6f7e0dcaa3c4ecd9658ee15fb3594ad: Bootstrap starting.
I20260929 15:51:26.347862 1556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6f7e0dcaa3c4ecd9658ee15fb3594ad: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:26.348832 1556 log.cc:824] T 00000000000000000000000000000000 P e6f7e0dcaa3c4ecd9658ee15fb3594ad: Log is configured to *not* fsync() on all Append() calls
Sep 29 15:51:26 dist-test-slave-r0ck krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:51:26.351989 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6f7e0dcaa3c4ecd9658ee15fb3594ad: No bootstrap required, opened a new log
I20260929 15:51:26.358661 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:26.336453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44526 (local address 127.0.97.62:33219)
0929 15:51:26.337274 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:26.337312 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:26.337381 (+ 69us) server_negotiation.cc:407] Connection header received
0929 15:51:26.339741 (+ 2360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:26.339768 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:26.340246 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:26.340676 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:26.342012 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:26.343901 (+ 1889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:26.345170 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:26.346279 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:26.346469 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:26.350560 (+ 4091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:26.350569 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:26.350586 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:26.350624 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:26.354477 (+ 3853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:26.355099 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:26.355110 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:26.355120 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:26.355318 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:26.355631 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:26.355636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:26.355640 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:26.356314 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:26.356593 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:26.356895 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 15:51:26.357319 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":349,"threads_started":1}
I20260929 15:51:26.362076 1556 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6f7e0dcaa3c4ecd9658ee15fb3594ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6f7e0dcaa3c4ecd9658ee15fb3594ad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33219 } }
I20260929 15:51:26.362375 1556 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6f7e0dcaa3c4ecd9658ee15fb3594ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:26.362443 1556 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6f7e0dcaa3c4ecd9658ee15fb3594ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6f7e0dcaa3c4ecd9658ee15fb3594ad, State: Initialized, Role: FOLLOWER
I20260929 15:51:26.362972 1556 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6f7e0dcaa3c4ecd9658ee15fb3594ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6f7e0dcaa3c4ecd9658ee15fb3594ad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33219 } }
I20260929 15:51:26.363242 1556 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6f7e0dcaa3c4ecd9658ee15fb3594ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:51:26.363407 1556 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6f7e0dcaa3c4ecd9658ee15fb3594ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:51:26.363591 1556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6f7e0dcaa3c4ecd9658ee15fb3594ad [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:26.365685 1556 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6f7e0dcaa3c4ecd9658ee15fb3594ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6f7e0dcaa3c4ecd9658ee15fb3594ad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33219 } }
I20260929 15:51:26.366137 1556 leader_election.cc:302] T 00000000000000000000000000000000 P e6f7e0dcaa3c4ecd9658ee15fb3594ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6f7e0dcaa3c4ecd9658ee15fb3594ad; no voters:
I20260929 15:51:26.366672 1556 leader_election.cc:288] T 00000000000000000000000000000000 P e6f7e0dcaa3c4ecd9658ee15fb3594ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:51:26.366796 1561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6f7e0dcaa3c4ecd9658ee15fb3594ad [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:51:26.367851 1561 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6f7e0dcaa3c4ecd9658ee15fb3594ad [term 1 LEADER]: Becoming Leader. State: Replica: e6f7e0dcaa3c4ecd9658ee15fb3594ad, State: Running, Role: LEADER
I20260929 15:51:26.368338 1561 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6f7e0dcaa3c4ecd9658ee15fb3594ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6f7e0dcaa3c4ecd9658ee15fb3594ad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33219 } }
I20260929 15:51:26.368695 1556 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6f7e0dcaa3c4ecd9658ee15fb3594ad [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:51:26.371542 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6f7e0dcaa3c4ecd9658ee15fb3594ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6f7e0dcaa3c4ecd9658ee15fb3594ad. Latest consensus state: current_term: 1 leader_uuid: "e6f7e0dcaa3c4ecd9658ee15fb3594ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6f7e0dcaa3c4ecd9658ee15fb3594ad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33219 } } }
I20260929 15:51:26.371518 1562 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6f7e0dcaa3c4ecd9658ee15fb3594ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6f7e0dcaa3c4ecd9658ee15fb3594ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6f7e0dcaa3c4ecd9658ee15fb3594ad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33219 } } }
I20260929 15:51:26.371930 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6f7e0dcaa3c4ecd9658ee15fb3594ad [sys.catalog]: This master's current role is: LEADER
I20260929 15:51:26.371938 1562 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6f7e0dcaa3c4ecd9658ee15fb3594ad [sys.catalog]: This master's current role is: LEADER
I20260929 15:51:26.374468 1568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:51:26.378470 1568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:51:26.386755 1568 catalog_manager.cc:1347] Generated new cluster ID: cbc3376a79c4467ba5dc7fadc7efb99f
I20260929 15:51:26.386865 1568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:51:26.394637 1568 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:51:26.396363 1568 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:51:26.404417 1568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6f7e0dcaa3c4ecd9658ee15fb3594ad: Generated new TSK 0
I20260929 15:51:26.405319 1568 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 15:51:26.449115 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33219
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:51:26.761094 1582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:51:26.761425 1582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:51:26.761600 1582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:51:26.768878 1582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:51:26.769009 1582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:51:26.769163 1582 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 15:51:26.769300 1582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:51:26.778946 1582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:33219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:51:26.780156 1582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:51:26.781466 1582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:51:26.794607 1587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:26.794523 1590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:26.794556 1588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:51:26.795521 1582 server_base.cc:1034] running on GCE node
I20260929 15:51:26.796120 1582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:51:26.796940 1582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:51:26.798182 1582 hybrid_clock.cc:648] HybridClock initialized: now 1790697086798130 us; error 49 us; skew 500 ppm
I20260929 15:51:26.798472 1582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:51:26 dist-test-slave-r0ck krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:51:26.801703 1582 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:51:26.803072 1582 webserver.cc:460] Webserver started at http://127.0.97.1:38003/ using document root <none> and password file <none>
I20260929 15:51:26.803723 1582 fs_manager.cc:359] Metadata directory not provided
I20260929 15:51:26.803851 1582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:51:26.804168 1582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:51:26.807031 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-0/data/instance:
uuid: "36dde0a7a9114639bc27dca417b288cc"
format_stamp: "Formatted at 2026-09-29 15:51:26 on dist-test-slave-r0ck"
server_key: "c7d91500d2d6a84a5c22e854e9ec9178"
server_key_iv: "7c7107245957309279f153f1ffa591ca"
server_key_version: "encryptionkey@0"
I20260929 15:51:26.807791 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "36dde0a7a9114639bc27dca417b288cc"
format_stamp: "Formatted at 2026-09-29 15:51:26 on dist-test-slave-r0ck"
server_key: "c7d91500d2d6a84a5c22e854e9ec9178"
server_key_iv: "7c7107245957309279f153f1ffa591ca"
server_key_version: "encryptionkey@0"
I20260929 15:51:26.812748 1582 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 15:51:26.816198 1597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:26.817471 1582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:51:26.817632 1582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-0/wal
uuid: "36dde0a7a9114639bc27dca417b288cc"
format_stamp: "Formatted at 2026-09-29 15:51:26 on dist-test-slave-r0ck"
server_key: "c7d91500d2d6a84a5c22e854e9ec9178"
server_key_iv: "7c7107245957309279f153f1ffa591ca"
server_key_version: "encryptionkey@0"
I20260929 15:51:26.817847 1582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:51:26.837720 1582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:51:26.841900 1582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:51:26.842202 1582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:51:26.844725 1582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:51:26.844833 1582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:26.845005 1582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:51:26.845177 1582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 29 15:51:26 dist-test-slave-r0ck krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:51:26.868381 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:26.845331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46881 (local address 127.0.97.62:33219)
0929 15:51:26.845526 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:26.845531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:26.846996 (+ 1465us) server_negotiation.cc:407] Connection header received
0929 15:51:26.850025 (+ 3029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:26.850032 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:26.850133 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:26.850273 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:26.852513 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:26.853472 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:26.855936 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:26.856734 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:26.856894 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:26.861167 (+ 4273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:26.861175 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:26.861180 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:26.861219 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:26.864913 (+ 3694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:26.865750 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:26.865758 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:26.865764 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:26.865943 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:26.866363 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:26.866371 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:26.866377 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:26.866812 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:26.866977 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:26.867618 (+ 641us) server_negotiation.cc:299] Negotiation successful
0929 15:51:26.867903 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 15:51:26.869143 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:26.845930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33219 (local address 127.0.97.1:46881)
0929 15:51:26.846794 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:26.846854 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:26.849602 (+ 2748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:26.850693 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:26.850707 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:26.851547 (+ 840us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:51:26.852339 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:26.852361 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:26.853681 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:26.853685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:26.855775 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:26.855787 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:26.857031 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:26.857039 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:26.857434 (+ 395us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:26.858240 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:51:26.858280 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:51:26.860930 (+ 2650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:51:26.865080 (+ 4150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:26.865090 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:26.865115 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:26.865627 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:26.866060 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:26.866063 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:26.866066 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:26.866245 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:26.866951 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:51:26.866957 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:51:26.867247 (+ 290us) client_negotiation.cc:769] Sending connection context
0929 15:51:26.867466 (+ 219us) client_negotiation.cc:240] Negotiation successful
0929 15:51:26.867797 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":341,"threads_started":1}
I20260929 15:51:26.890417 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:26.876027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33121 (local address 127.0.97.62:33219)
0929 15:51:26.876211 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:26.876216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:26.877056 (+ 840us) server_negotiation.cc:407] Connection header received
0929 15:51:26.877164 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:26.877171 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:26.877263 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:26.877383 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:26.878796 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:26.879563 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:26.880872 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:26.881242 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:26.881409 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:26.883564 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:26.883569 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:26.883572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:26.883600 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:26.887585 (+ 3985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:26.888193 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:26.888201 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:26.888207 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:26.888390 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:26.888728 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:26.888736 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:26.888742 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:26.889158 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:26.889325 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:26.889877 (+ 552us) server_negotiation.cc:299] Negotiation successful
0929 15:51:26.890078 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 15:51:26.890553 1664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:26.876101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33219 (local address 127.0.97.1:33121)
0929 15:51:26.876944 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:26.876969 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:26.877073 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:26.877539 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:26.877546 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:26.878158 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:51:26.878656 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:26.878667 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:26.879716 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:26.879723 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:26.880711 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:26.880721 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:26.881450 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:26.881457 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:26.881633 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:26.882387 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:51:26.882403 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:51:26.883311 (+ 908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:51:26.887722 (+ 4411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:26.887725 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:26.887728 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:26.888080 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:26.888514 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:26.888517 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:26.888519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:26.888628 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:26.889284 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:51:26.889288 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:51:26.889570 (+ 282us) client_negotiation.cc:769] Sending connection context
0929 15:51:26.889752 (+ 182us) client_negotiation.cc:240] Negotiation successful
0929 15:51:26.890079 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":549,"threads_started":1}
I20260929 15:51:26.901033 1582 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46659
I20260929 15:51:26.901110 1716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46659 every 8 connection(s)
I20260929 15:51:26.902269 1582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:51:26.904196 1717 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33219
I20260929 15:51:26.904425 1717 heartbeater.cc:464] Registering TS with master...
I20260929 15:51:26.905030 1717 heartbeater.cc:511] Master 127.0.97.62:33219 requested a full tablet report, sending...
I20260929 15:51:26.906277 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 1582
I20260929 15:51:26.906489 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:51:26.907092 388 external_mini_cluster.cc:1428] Setting key edf33f2af8fc82607608c27ec3c6bb52
I20260929 15:51:26.907084 1511 ts_manager.cc:194] Registered new tserver with Master: 36dde0a7a9114639bc27dca417b288cc (127.0.97.1:46659)
I20260929 15:51:26.908483 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46881
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 15:51:26.943877 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33219
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:51:27.253767 1721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:51:27.254024 1721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:51:27.254181 1721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:51:27.261595 1721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:51:27.261693 1721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:51:27.261819 1721 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 15:51:27.261946 1721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:51:27.271756 1721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:33219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:51:27.272979 1721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:51:27.274278 1721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:51:27.286897 1729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:27.286993 1726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:27.286998 1727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:51:27.287897 1721 server_base.cc:1034] running on GCE node
I20260929 15:51:27.288465 1721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:51:27.289310 1721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:51:27.290565 1721 hybrid_clock.cc:648] HybridClock initialized: now 1790697087290513 us; error 32 us; skew 500 ppm
I20260929 15:51:27.290841 1721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:51:27 dist-test-slave-r0ck krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:51:27.298420 1721 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:51:27.299829 1721 webserver.cc:460] Webserver started at http://127.0.97.2:45851/ using document root <none> and password file <none>
I20260929 15:51:27.300514 1721 fs_manager.cc:359] Metadata directory not provided
I20260929 15:51:27.300644 1721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:51:27.300961 1721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:51:27.304649 1721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-1/data/instance:
uuid: "8f77615a1c5e4961862b33a057d44c3c"
format_stamp: "Formatted at 2026-09-29 15:51:27 on dist-test-slave-r0ck"
server_key: "ed2360204d34d6ef2abe3499de86b022"
server_key_iv: "79e798b029480ccccbaf984d4422f68d"
server_key_version: "encryptionkey@0"
I20260929 15:51:27.305611 1721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8f77615a1c5e4961862b33a057d44c3c"
format_stamp: "Formatted at 2026-09-29 15:51:27 on dist-test-slave-r0ck"
server_key: "ed2360204d34d6ef2abe3499de86b022"
server_key_iv: "79e798b029480ccccbaf984d4422f68d"
server_key_version: "encryptionkey@0"
I20260929 15:51:27.310449 1721 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 15:51:27.313862 1736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:27.315116 1721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 15:51:27.315281 1721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-1/wal
uuid: "8f77615a1c5e4961862b33a057d44c3c"
format_stamp: "Formatted at 2026-09-29 15:51:27 on dist-test-slave-r0ck"
server_key: "ed2360204d34d6ef2abe3499de86b022"
server_key_iv: "79e798b029480ccccbaf984d4422f68d"
server_key_version: "encryptionkey@0"
I20260929 15:51:27.315487 1721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:51:27.342963 1721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:51:27.346706 1721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:51:27.346976 1721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:51:27.349565 1721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:51:27.349663 1721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:27.349807 1721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:51:27.349879 1721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 15:51:27 dist-test-slave-r0ck krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:51:27.373955 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:27.350171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40841 (local address 127.0.97.62:33219)
0929 15:51:27.350408 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:27.350416 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:27.351531 (+ 1115us) server_negotiation.cc:407] Connection header received
0929 15:51:27.353677 (+ 2146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:27.353683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:27.353772 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:27.353884 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:27.355885 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:27.356971 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:27.360103 (+ 3132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:27.360646 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:27.360781 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:27.366369 (+ 5588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:27.366374 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:27.366377 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:27.366407 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:27.370276 (+ 3869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:27.371109 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:27.371118 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:27.371125 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:27.371299 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:27.371798 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:27.371806 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:27.371812 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:27.372253 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:27.372442 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:27.373108 (+ 666us) server_negotiation.cc:299] Negotiation successful
0929 15:51:27.373437 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 15:51:27.374702 1745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:27.350628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33219 (local address 127.0.97.2:40841)
0929 15:51:27.351379 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:27.351413 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:27.353342 (+ 1929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:27.354137 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:27.354155 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:27.354855 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:51:27.355701 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:27.355736 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:27.357155 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:27.357164 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:27.359934 (+ 2770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:27.359958 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:27.361215 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:27.361229 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:27.361523 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:27.362379 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:51:27.362403 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:51:27.366073 (+ 3670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:51:27.370435 (+ 4362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:27.370441 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:27.370482 (+ 41us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:27.370973 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:27.371406 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:27.371409 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:27.371411 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:27.371656 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:27.372386 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:51:27.372392 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:51:27.372714 (+ 322us) client_negotiation.cc:769] Sending connection context
0929 15:51:27.372945 (+ 231us) client_negotiation.cc:240] Negotiation successful
0929 15:51:27.373450 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":341,"threads_started":1}
I20260929 15:51:27.397182 1721 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34683
I20260929 15:51:27.397184 1855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34683 every 8 connection(s)
I20260929 15:51:27.398063 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:27.380594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50235 (local address 127.0.97.62:33219)
0929 15:51:27.380793 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:27.380801 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:27.381761 (+ 960us) server_negotiation.cc:407] Connection header received
0929 15:51:27.381877 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:27.381885 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:27.381981 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:27.382084 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:27.383786 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:27.384688 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:27.387431 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:27.388201 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:27.388336 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:27.391163 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:27.391172 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:27.391179 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:27.391228 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:27.394927 (+ 3699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:27.395811 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:27.395816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:27.395818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:27.395975 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:27.396491 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:27.396497 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:27.396500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:27.396926 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:27.397072 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:27.397661 (+ 589us) server_negotiation.cc:299] Negotiation successful
0929 15:51:27.397820 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 15:51:27.398365 1808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:27.380602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33219 (local address 127.0.97.2:50235)
0929 15:51:27.381656 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:27.381680 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:27.381781 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:27.382281 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:27.382291 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:27.383035 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:51:27.383552 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:27.383568 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:27.384825 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:27.384834 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:27.387192 (+ 2358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:27.387211 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:27.388606 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:27.388612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:27.388773 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:27.389538 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:51:27.389564 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:51:27.390935 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:51:27.395110 (+ 4175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:27.395115 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:27.395119 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:27.395690 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:27.396106 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:27.396110 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:27.396114 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:27.396327 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:27.397150 (+ 823us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:51:27.397160 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:51:27.397404 (+ 244us) client_negotiation.cc:769] Sending connection context
0929 15:51:27.397551 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 15:51:27.397840 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":916,"thread_start_us":765,"threads_started":1}
I20260929 15:51:27.398824 1721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:51:27.399703 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 1721
I20260929 15:51:27.399951 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:51:27.400616 388 external_mini_cluster.cc:1428] Setting key c7094a0a671efcc500941eb3f4ac9a08
I20260929 15:51:27.401966 1856 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33219
I20260929 15:51:27.402910 1856 heartbeater.cc:464] Registering TS with master...
I20260929 15:51:27.403503 1856 heartbeater.cc:511] Master 127.0.97.62:33219 requested a full tablet report, sending...
I20260929 15:51:27.404816 1511 ts_manager.cc:194] Registered new tserver with Master: 8f77615a1c5e4961862b33a057d44c3c (127.0.97.2:34683)
I20260929 15:51:27.405613 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:40841
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 15:51:27.437701 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33219
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:51:27.743891 1860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:51:27.744186 1860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:51:27.744314 1860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:51:27.751370 1860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:51:27.751468 1860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:51:27.751652 1860 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 15:51:27.751770 1860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:51:27.761123 1860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:33219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:51:27.762349 1860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:51:27.763649 1860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:51:27.776134 1865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:27.776257 1868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:27.776084 1866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:51:27.777652 1860 server_base.cc:1034] running on GCE node
I20260929 15:51:27.778221 1860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:51:27.779068 1860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:51:27.780339 1860 hybrid_clock.cc:648] HybridClock initialized: now 1790697087780275 us; error 43 us; skew 500 ppm
I20260929 15:51:27.780655 1860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:51:27 dist-test-slave-r0ck krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:51:27.783656 1860 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:51:27.784999 1860 webserver.cc:460] Webserver started at http://127.0.97.3:33271/ using document root <none> and password file <none>
I20260929 15:51:27.785742 1860 fs_manager.cc:359] Metadata directory not provided
I20260929 15:51:27.785825 1860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:51:27.786167 1860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:51:27.788945 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-2/data/instance:
uuid: "f9dce3d411234778a20049dc1d077dd2"
format_stamp: "Formatted at 2026-09-29 15:51:27 on dist-test-slave-r0ck"
server_key: "7d64489a6956a97ba61076c4b88b418f"
server_key_iv: "ca09c0931898c0b0f356878cacded6a3"
server_key_version: "encryptionkey@0"
I20260929 15:51:27.789703 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f9dce3d411234778a20049dc1d077dd2"
format_stamp: "Formatted at 2026-09-29 15:51:27 on dist-test-slave-r0ck"
server_key: "7d64489a6956a97ba61076c4b88b418f"
server_key_iv: "ca09c0931898c0b0f356878cacded6a3"
server_key_version: "encryptionkey@0"
I20260929 15:51:27.794498 1860 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 15:51:27.797922 1875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:27.799132 1860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:51:27.799289 1860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-2/wal
uuid: "f9dce3d411234778a20049dc1d077dd2"
format_stamp: "Formatted at 2026-09-29 15:51:27 on dist-test-slave-r0ck"
server_key: "7d64489a6956a97ba61076c4b88b418f"
server_key_iv: "ca09c0931898c0b0f356878cacded6a3"
server_key_version: "encryptionkey@0"
I20260929 15:51:27.799494 1860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:51:27.826490 1860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:51:27.830286 1860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:51:27.830554 1860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:51:27.833171 1860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:51:27.833283 1860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:27.833447 1860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:51:27.833513 1860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 15:51:27 dist-test-slave-r0ck krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:51:27.855655 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:27.833588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42781 (local address 127.0.97.62:33219)
0929 15:51:27.833804 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:27.833812 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:27.835107 (+ 1295us) server_negotiation.cc:407] Connection header received
0929 15:51:27.837025 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:27.837031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:27.837122 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:27.837230 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:27.839173 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:27.840221 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:27.843515 (+ 3294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:27.844074 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:27.844215 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:27.849597 (+ 5382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:27.849602 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:27.849606 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:27.849639 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:27.852172 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:27.852915 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:27.852919 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:27.852921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:27.853039 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:27.853560 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:27.853571 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:27.853573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:27.853961 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:27.854112 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:27.855039 (+ 927us) server_negotiation.cc:299] Negotiation successful
0929 15:51:27.855264 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 15:51:27.856787 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:27.834213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33219 (local address 127.0.97.3:42781)
0929 15:51:27.834951 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:27.834991 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:27.836749 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:27.837484 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:27.837492 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:27.838196 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:51:27.838993 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:27.839023 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:27.840436 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:27.840445 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:27.843366 (+ 2921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:27.843381 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:27.844358 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:27.844368 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:27.844734 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:27.845635 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:51:27.845659 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:51:27.849290 (+ 3631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:51:27.852320 (+ 3030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:27.852329 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:27.852347 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:27.852767 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:27.853153 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:27.853158 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:27.853161 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:27.853432 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:27.854099 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:51:27.854107 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:51:27.854585 (+ 478us) client_negotiation.cc:769] Sending connection context
0929 15:51:27.854952 (+ 367us) client_negotiation.cc:240] Negotiation successful
0929 15:51:27.855493 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":322,"threads_started":1}
I20260929 15:51:27.878772 1946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:27.864422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33219 (local address 127.0.97.3:38163)
0929 15:51:27.865045 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:27.865063 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:27.865214 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:27.865602 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:27.865610 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:27.866400 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:51:27.866913 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:27.866927 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:27.868069 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:27.868075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:27.869230 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:27.869247 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:27.870310 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:27.870317 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:27.870488 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:27.871211 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:51:27.871227 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:51:27.872074 (+ 847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:51:27.875510 (+ 3436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:27.875515 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:27.875519 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:27.875926 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:27.876442 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:27.876445 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:27.876448 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:27.876560 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:27.877196 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:51:27.877208 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:51:27.877479 (+ 271us) client_negotiation.cc:769] Sending connection context
0929 15:51:27.877660 (+ 181us) client_negotiation.cc:240] Negotiation successful
0929 15:51:27.878086 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":359,"threads_started":1}
I20260929 15:51:27.880088 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:27.864455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38163 (local address 127.0.97.62:33219)
0929 15:51:27.864606 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:27.864610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:27.865152 (+ 542us) server_negotiation.cc:407] Connection header received
0929 15:51:27.865389 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:27.865396 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:27.865470 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:27.865574 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:27.867062 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:27.867941 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:27.869429 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:27.870102 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:27.870251 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:27.872254 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:27.872261 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:27.872265 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:27.872298 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:27.875312 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:27.876080 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:27.876089 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:27.876095 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:27.876313 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:27.876668 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:27.876672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:27.876676 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:27.877033 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:27.879089 (+ 2056us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:27.879255 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 15:51:27.879524 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 15:51:27.889274 1994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42325 every 8 connection(s)
I20260929 15:51:27.889271 1860 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42325
I20260929 15:51:27.890484 1860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:51:27.891912 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 1860
I20260929 15:51:27.892103 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:51:27.892529 388 external_mini_cluster.cc:1428] Setting key 574e62b0437c83518c3a5cee92a16ba5
I20260929 15:51:27.893183 1995 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33219
I20260929 15:51:27.893436 1995 heartbeater.cc:464] Registering TS with master...
I20260929 15:51:27.894230 1995 heartbeater.cc:511] Master 127.0.97.62:33219 requested a full tablet report, sending...
I20260929 15:51:27.895627 1511 ts_manager.cc:194] Registered new tserver with Master: f9dce3d411234778a20049dc1d077dd2 (127.0.97.3:42325)
I20260929 15:51:27.896446 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42781
I20260929 15:51:27.907486 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:51:27.910996 1717 heartbeater.cc:503] Master 127.0.97.62:33219 was elected leader, sending a full tablet report...
Sep 29 15:51:27 dist-test-slave-r0ck krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697087, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 15:51:27 dist-test-slave-r0ck krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697087, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:51:27.951598 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:27.938180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43342 (local address 127.0.97.62:33219)
0929 15:51:27.938361 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:27.938366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:27.938698 (+ 332us) server_negotiation.cc:407] Connection header received
0929 15:51:27.938839 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:27.938843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:27.938904 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:27.939000 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:27.940287 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:27.941608 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:27.942823 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:27.943240 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:27.943326 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:27.946986 (+ 3660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:27.946991 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:27.946994 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:27.947028 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:27.949509 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:27.950077 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:27.950081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:27.950083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:27.950189 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:27.950510 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:27.950513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:27.950515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:27.950759 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:27.950942 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:27.951215 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 15:51:27.951367 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 15:51:27.955575 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43342:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:51:27.962357 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:51:27.991199 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:27.974644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34156 (local address 127.0.97.2:34683)
0929 15:51:27.982482 (+ 7838us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:27.982489 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:27.982512 (+ 23us) server_negotiation.cc:407] Connection header received
0929 15:51:27.982592 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:27.982599 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:27.982791 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:27.983007 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:51:27.983690 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:27.985122 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:27.987843 (+ 2721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:27.989316 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:27.989475 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:27.989672 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:27.989983 (+ 311us) server_negotiation.cc:299] Negotiation successful
0929 15:51:27.990168 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7698,"thread_start_us":7576,"threads_started":1}
I20260929 15:51:27.990798 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:27.974278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34683 (local address 127.0.0.1:34156)
0929 15:51:27.976985 (+ 2707us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:27.977005 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:27.977132 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:27.982999 (+ 5867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:27.983006 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:27.983036 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:51:27.983487 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:27.983499 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:27.985295 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:27.985302 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:27.987659 (+ 2357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:27.987674 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:27.989529 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:27.989540 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:27.989739 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:27.989788 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 15:51:27.989929 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 15:51:27.990118 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2543,"thread_start_us":9496,"threads_started":1}
I20260929 15:51:27.991906 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:27.974234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42325 (local address 127.0.0.1:40016)
0929 15:51:27.975669 (+ 1435us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:27.975704 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:27.975862 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:27.984272 (+ 8410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:27.984276 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:27.984301 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:51:27.984649 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:27.984661 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:27.987260 (+ 2599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:27.987267 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:27.989024 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:27.989035 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:27.991168 (+ 2133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:27.991173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:27.991340 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:27.991379 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 15:51:27.991448 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 15:51:27.991575 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1217,"mutex_wait_us":3,"thread_start_us":1080,"threads_started":1}
I20260929 15:51:27.992802 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:27.974782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40016 (local address 127.0.97.3:42325)
0929 15:51:27.983694 (+ 8912us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:27.983702 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:27.983730 (+ 28us) server_negotiation.cc:407] Connection header received
0929 15:51:27.983820 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:27.983826 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:27.984068 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:27.984831 (+ 763us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:51:27.985316 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:27.987090 (+ 1774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:27.989187 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:27.990936 (+ 1749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:27.992099 (+ 1163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:27.992291 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:27.992352 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 15:51:27.992456 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8738,"thread_start_us":8549,"threads_started":1}
I20260929 15:51:27.992832 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:27.976246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46659 (local address 127.0.0.1:42208)
0929 15:51:27.984029 (+ 7783us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:27.984044 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:27.984183 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:27.984644 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:27.984649 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:27.984673 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:51:27.985041 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:27.985050 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:27.987827 (+ 2777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:27.987833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:27.989908 (+ 2075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:27.989923 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:27.992196 (+ 2273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:27.992202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:27.992367 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:27.992397 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 15:51:27.992490 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 15:51:27.992611 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7661,"thread_start_us":431,"threads_started":1}
I20260929 15:51:27.993649 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:27.976368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42208 (local address 127.0.97.1:46659)
0929 15:51:27.978050 (+ 1682us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:27.978064 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:27.984148 (+ 6084us) server_negotiation.cc:407] Connection header received
0929 15:51:27.984317 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:27.984323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:27.984474 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:27.984657 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:51:27.985202 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:27.986771 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:27.990089 (+ 3318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:27.991509 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:27.991631 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:27.991814 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:27.993170 (+ 1356us) server_negotiation.cc:299] Negotiation successful
0929 15:51:27.993332 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1502,"thread_start_us":1393,"threads_started":1}
I20260929 15:51:27.995373 1786 tablet_service.cc:1467] Processing CreateTablet for tablet dcfb01a790f240708dfc9cc1b493eaff (DEFAULT_TABLE table=test-table [id=0631e37203dd46dfae1270bb821558f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:51:27.996510 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcfb01a790f240708dfc9cc1b493eaff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:27.995994 1925 tablet_service.cc:1467] Processing CreateTablet for tablet dcfb01a790f240708dfc9cc1b493eaff (DEFAULT_TABLE table=test-table [id=0631e37203dd46dfae1270bb821558f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:51:27.996799 1647 tablet_service.cc:1467] Processing CreateTablet for tablet dcfb01a790f240708dfc9cc1b493eaff (DEFAULT_TABLE table=test-table [id=0631e37203dd46dfae1270bb821558f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:51:27.997478 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcfb01a790f240708dfc9cc1b493eaff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:27.998096 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcfb01a790f240708dfc9cc1b493eaff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:28.007726 2014 tablet_bootstrap.cc:492] T dcfb01a790f240708dfc9cc1b493eaff P 8f77615a1c5e4961862b33a057d44c3c: Bootstrap starting.
I20260929 15:51:28.010792 2014 tablet_bootstrap.cc:654] T dcfb01a790f240708dfc9cc1b493eaff P 8f77615a1c5e4961862b33a057d44c3c: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:28.011816 2015 tablet_bootstrap.cc:492] T dcfb01a790f240708dfc9cc1b493eaff P 36dde0a7a9114639bc27dca417b288cc: Bootstrap starting.
I20260929 15:51:28.012136 2014 log.cc:824] T dcfb01a790f240708dfc9cc1b493eaff P 8f77615a1c5e4961862b33a057d44c3c: Log is configured to *not* fsync() on all Append() calls
I20260929 15:51:28.012753 2016 tablet_bootstrap.cc:492] T dcfb01a790f240708dfc9cc1b493eaff P f9dce3d411234778a20049dc1d077dd2: Bootstrap starting.
I20260929 15:51:28.016088 2016 tablet_bootstrap.cc:654] T dcfb01a790f240708dfc9cc1b493eaff P f9dce3d411234778a20049dc1d077dd2: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:28.016830 2015 tablet_bootstrap.cc:654] T dcfb01a790f240708dfc9cc1b493eaff P 36dde0a7a9114639bc27dca417b288cc: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:28.017387 2016 log.cc:824] T dcfb01a790f240708dfc9cc1b493eaff P f9dce3d411234778a20049dc1d077dd2: Log is configured to *not* fsync() on all Append() calls
I20260929 15:51:28.018098 2015 log.cc:824] T dcfb01a790f240708dfc9cc1b493eaff P 36dde0a7a9114639bc27dca417b288cc: Log is configured to *not* fsync() on all Append() calls
I20260929 15:51:28.019227 2014 tablet_bootstrap.cc:492] T dcfb01a790f240708dfc9cc1b493eaff P 8f77615a1c5e4961862b33a057d44c3c: No bootstrap required, opened a new log
I20260929 15:51:28.019562 2014 ts_tablet_manager.cc:1397] T dcfb01a790f240708dfc9cc1b493eaff P 8f77615a1c5e4961862b33a057d44c3c: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260929 15:51:28.020018 2016 tablet_bootstrap.cc:492] T dcfb01a790f240708dfc9cc1b493eaff P f9dce3d411234778a20049dc1d077dd2: No bootstrap required, opened a new log
I20260929 15:51:28.020294 2016 ts_tablet_manager.cc:1397] T dcfb01a790f240708dfc9cc1b493eaff P f9dce3d411234778a20049dc1d077dd2: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 15:51:28.023816 2015 tablet_bootstrap.cc:492] T dcfb01a790f240708dfc9cc1b493eaff P 36dde0a7a9114639bc27dca417b288cc: No bootstrap required, opened a new log
I20260929 15:51:28.024052 2015 ts_tablet_manager.cc:1397] T dcfb01a790f240708dfc9cc1b493eaff P 36dde0a7a9114639bc27dca417b288cc: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260929 15:51:28.027878 2016 raft_consensus.cc:348] T dcfb01a790f240708dfc9cc1b493eaff P f9dce3d411234778a20049dc1d077dd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dce3d411234778a20049dc1d077dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42325 } } peers { permanent_uuid: "8f77615a1c5e4961862b33a057d44c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34683 } } peers { permanent_uuid: "36dde0a7a9114639bc27dca417b288cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46659 } }
I20260929 15:51:28.028230 2016 raft_consensus.cc:374] T dcfb01a790f240708dfc9cc1b493eaff P f9dce3d411234778a20049dc1d077dd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:28.028337 2016 raft_consensus.cc:729] T dcfb01a790f240708dfc9cc1b493eaff P f9dce3d411234778a20049dc1d077dd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9dce3d411234778a20049dc1d077dd2, State: Initialized, Role: FOLLOWER
I20260929 15:51:28.028965 2016 consensus_queue.cc:260] T dcfb01a790f240708dfc9cc1b493eaff P f9dce3d411234778a20049dc1d077dd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dce3d411234778a20049dc1d077dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42325 } } peers { permanent_uuid: "8f77615a1c5e4961862b33a057d44c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34683 } } peers { permanent_uuid: "36dde0a7a9114639bc27dca417b288cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46659 } }
I20260929 15:51:28.030526 2014 raft_consensus.cc:348] T dcfb01a790f240708dfc9cc1b493eaff P 8f77615a1c5e4961862b33a057d44c3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dce3d411234778a20049dc1d077dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42325 } } peers { permanent_uuid: "8f77615a1c5e4961862b33a057d44c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34683 } } peers { permanent_uuid: "36dde0a7a9114639bc27dca417b288cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46659 } }
I20260929 15:51:28.030923 2014 raft_consensus.cc:374] T dcfb01a790f240708dfc9cc1b493eaff P 8f77615a1c5e4961862b33a057d44c3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:28.031025 2014 raft_consensus.cc:729] T dcfb01a790f240708dfc9cc1b493eaff P 8f77615a1c5e4961862b33a057d44c3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f77615a1c5e4961862b33a057d44c3c, State: Initialized, Role: FOLLOWER
I20260929 15:51:28.031690 1995 heartbeater.cc:503] Master 127.0.97.62:33219 was elected leader, sending a full tablet report...
I20260929 15:51:28.032214 2016 ts_tablet_manager.cc:1428] T dcfb01a790f240708dfc9cc1b493eaff P f9dce3d411234778a20049dc1d077dd2: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20260929 15:51:28.032531 2014 consensus_queue.cc:260] T dcfb01a790f240708dfc9cc1b493eaff P 8f77615a1c5e4961862b33a057d44c3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dce3d411234778a20049dc1d077dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42325 } } peers { permanent_uuid: "8f77615a1c5e4961862b33a057d44c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34683 } } peers { permanent_uuid: "36dde0a7a9114639bc27dca417b288cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46659 } }
I20260929 15:51:28.033965 1856 heartbeater.cc:503] Master 127.0.97.62:33219 was elected leader, sending a full tablet report...
I20260929 15:51:28.034623 2014 ts_tablet_manager.cc:1428] T dcfb01a790f240708dfc9cc1b493eaff P 8f77615a1c5e4961862b33a057d44c3c: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
I20260929 15:51:28.034490 2015 raft_consensus.cc:348] T dcfb01a790f240708dfc9cc1b493eaff P 36dde0a7a9114639bc27dca417b288cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dce3d411234778a20049dc1d077dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42325 } } peers { permanent_uuid: "8f77615a1c5e4961862b33a057d44c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34683 } } peers { permanent_uuid: "36dde0a7a9114639bc27dca417b288cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46659 } }
I20260929 15:51:28.034861 2015 raft_consensus.cc:374] T dcfb01a790f240708dfc9cc1b493eaff P 36dde0a7a9114639bc27dca417b288cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:28.034953 2015 raft_consensus.cc:729] T dcfb01a790f240708dfc9cc1b493eaff P 36dde0a7a9114639bc27dca417b288cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36dde0a7a9114639bc27dca417b288cc, State: Initialized, Role: FOLLOWER
I20260929 15:51:28.035763 2015 consensus_queue.cc:260] T dcfb01a790f240708dfc9cc1b493eaff P 36dde0a7a9114639bc27dca417b288cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dce3d411234778a20049dc1d077dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42325 } } peers { permanent_uuid: "8f77615a1c5e4961862b33a057d44c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34683 } } peers { permanent_uuid: "36dde0a7a9114639bc27dca417b288cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46659 } }
I20260929 15:51:28.037388 2015 ts_tablet_manager.cc:1428] T dcfb01a790f240708dfc9cc1b493eaff P 36dde0a7a9114639bc27dca417b288cc: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
W20260929 15:51:28.142125 1996 tablet.cc:2369] T dcfb01a790f240708dfc9cc1b493eaff P f9dce3d411234778a20049dc1d077dd2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:51:28.153291 1857 tablet.cc:2369] T dcfb01a790f240708dfc9cc1b493eaff P 8f77615a1c5e4961862b33a057d44c3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:51:28.154688 1718 tablet.cc:2369] T dcfb01a790f240708dfc9cc1b493eaff P 36dde0a7a9114639bc27dca417b288cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:51:28.184077 2022 raft_consensus.cc:482] T dcfb01a790f240708dfc9cc1b493eaff P 36dde0a7a9114639bc27dca417b288cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:51:28.184394 2022 raft_consensus.cc:504] T dcfb01a790f240708dfc9cc1b493eaff P 36dde0a7a9114639bc27dca417b288cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dce3d411234778a20049dc1d077dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42325 } } peers { permanent_uuid: "8f77615a1c5e4961862b33a057d44c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34683 } } peers { permanent_uuid: "36dde0a7a9114639bc27dca417b288cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46659 } }
I20260929 15:51:28.186161 2022 leader_election.cc:288] T dcfb01a790f240708dfc9cc1b493eaff P 36dde0a7a9114639bc27dca417b288cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9dce3d411234778a20049dc1d077dd2 (127.0.97.3:42325), 8f77615a1c5e4961862b33a057d44c3c (127.0.97.2:34683)
I20260929 15:51:28.193222 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:28.186658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34339 (local address 127.0.97.3:42325)
0929 15:51:28.186909 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:28.186919 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:28.187920 (+ 1001us) server_negotiation.cc:407] Connection header received
0929 15:51:28.188177 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:28.188181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:28.188243 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:28.188345 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:51:28.188880 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:28.189637 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:28.191578 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:28.192325 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:28.192471 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:28.192583 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:28.192848 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 15:51:28.192975 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 15:51:28.193302 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:28.186644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42325 (local address 127.0.97.1:34339)
0929 15:51:28.187784 (+ 1140us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:28.187801 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:28.188022 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:28.188371 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:28.188375 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:28.188400 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:51:28.188724 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:28.188733 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:28.189762 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:28.189769 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:28.191420 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:28.191439 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:28.192545 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:28.192551 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:28.192692 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:28.192715 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 15:51:28.192824 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 15:51:28.192948 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1015,"thread_start_us":1108,"threads_started":1}
I20260929 15:51:28.193745 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:28.186702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43131 (local address 127.0.97.2:34683)
0929 15:51:28.186864 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:28.186895 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:28.188229 (+ 1334us) server_negotiation.cc:407] Connection header received
0929 15:51:28.188422 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:28.188428 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:28.188495 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:28.188598 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:51:28.189599 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:28.190271 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:28.192088 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:28.192829 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:28.192981 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:28.193085 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:28.193387 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 15:51:28.193499 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 15:51:28.193810 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:28.186644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34683 (local address 127.0.97.1:43131)
0929 15:51:28.188116 (+ 1472us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:28.188130 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:28.188277 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:28.188612 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:28.188620 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:28.188643 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:51:28.189453 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:28.189464 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:28.190403 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:28.190408 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:28.191945 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:28.191956 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:28.193055 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:28.193061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:28.193201 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:28.193224 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 15:51:28.193330 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 15:51:28.193461 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1356,"mutex_wait_us":292,"thread_start_us":520,"threads_started":1}
I20260929 15:51:28.194576 1950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcfb01a790f240708dfc9cc1b493eaff" candidate_uuid: "36dde0a7a9114639bc27dca417b288cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9dce3d411234778a20049dc1d077dd2" is_pre_election: true
I20260929 15:51:28.194703 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcfb01a790f240708dfc9cc1b493eaff" candidate_uuid: "36dde0a7a9114639bc27dca417b288cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f77615a1c5e4961862b33a057d44c3c" is_pre_election: true
I20260929 15:51:28.195182 1811 raft_consensus.cc:2454] T dcfb01a790f240708dfc9cc1b493eaff P 8f77615a1c5e4961862b33a057d44c3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36dde0a7a9114639bc27dca417b288cc in term 0.
I20260929 15:51:28.195182 1950 raft_consensus.cc:2454] T dcfb01a790f240708dfc9cc1b493eaff P f9dce3d411234778a20049dc1d077dd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36dde0a7a9114639bc27dca417b288cc in term 0.
I20260929 15:51:28.195896 1600 leader_election.cc:302] T dcfb01a790f240708dfc9cc1b493eaff P 36dde0a7a9114639bc27dca417b288cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 36dde0a7a9114639bc27dca417b288cc, 8f77615a1c5e4961862b33a057d44c3c, f9dce3d411234778a20049dc1d077dd2; no voters:
I20260929 15:51:28.196322 2022 raft_consensus.cc:2781] T dcfb01a790f240708dfc9cc1b493eaff P 36dde0a7a9114639bc27dca417b288cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:51:28.196489 2022 raft_consensus.cc:482] T dcfb01a790f240708dfc9cc1b493eaff P 36dde0a7a9114639bc27dca417b288cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:51:28.196620 2022 raft_consensus.cc:3037] T dcfb01a790f240708dfc9cc1b493eaff P 36dde0a7a9114639bc27dca417b288cc [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:28.199005 2022 raft_consensus.cc:504] T dcfb01a790f240708dfc9cc1b493eaff P 36dde0a7a9114639bc27dca417b288cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dce3d411234778a20049dc1d077dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42325 } } peers { permanent_uuid: "8f77615a1c5e4961862b33a057d44c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34683 } } peers { permanent_uuid: "36dde0a7a9114639bc27dca417b288cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46659 } }
I20260929 15:51:28.199563 2022 leader_election.cc:288] T dcfb01a790f240708dfc9cc1b493eaff P 36dde0a7a9114639bc27dca417b288cc [CANDIDATE]: Term 1 election: Requested vote from peers f9dce3d411234778a20049dc1d077dd2 (127.0.97.3:42325), 8f77615a1c5e4961862b33a057d44c3c (127.0.97.2:34683)
I20260929 15:51:28.199994 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcfb01a790f240708dfc9cc1b493eaff" candidate_uuid: "36dde0a7a9114639bc27dca417b288cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f77615a1c5e4961862b33a057d44c3c"
I20260929 15:51:28.200011 1950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcfb01a790f240708dfc9cc1b493eaff" candidate_uuid: "36dde0a7a9114639bc27dca417b288cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9dce3d411234778a20049dc1d077dd2"
I20260929 15:51:28.200191 1811 raft_consensus.cc:3037] T dcfb01a790f240708dfc9cc1b493eaff P 8f77615a1c5e4961862b33a057d44c3c [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:28.200203 1950 raft_consensus.cc:3037] T dcfb01a790f240708dfc9cc1b493eaff P f9dce3d411234778a20049dc1d077dd2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:28.202517 1950 raft_consensus.cc:2454] T dcfb01a790f240708dfc9cc1b493eaff P f9dce3d411234778a20049dc1d077dd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36dde0a7a9114639bc27dca417b288cc in term 1.
I20260929 15:51:28.202780 1811 raft_consensus.cc:2454] T dcfb01a790f240708dfc9cc1b493eaff P 8f77615a1c5e4961862b33a057d44c3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36dde0a7a9114639bc27dca417b288cc in term 1.
I20260929 15:51:28.202898 1599 leader_election.cc:302] T dcfb01a790f240708dfc9cc1b493eaff P 36dde0a7a9114639bc27dca417b288cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36dde0a7a9114639bc27dca417b288cc, f9dce3d411234778a20049dc1d077dd2; no voters:
I20260929 15:51:28.203218 2022 raft_consensus.cc:2781] T dcfb01a790f240708dfc9cc1b493eaff P 36dde0a7a9114639bc27dca417b288cc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:51:28.203630 2022 raft_consensus.cc:686] T dcfb01a790f240708dfc9cc1b493eaff P 36dde0a7a9114639bc27dca417b288cc [term 1 LEADER]: Becoming Leader. State: Replica: 36dde0a7a9114639bc27dca417b288cc, State: Running, Role: LEADER
I20260929 15:51:28.204142 2022 consensus_queue.cc:237] T dcfb01a790f240708dfc9cc1b493eaff P 36dde0a7a9114639bc27dca417b288cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dce3d411234778a20049dc1d077dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42325 } } peers { permanent_uuid: "8f77615a1c5e4961862b33a057d44c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34683 } } peers { permanent_uuid: "36dde0a7a9114639bc27dca417b288cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46659 } }
I20260929 15:51:28.208433 1511 catalog_manager.cc:5495] T dcfb01a790f240708dfc9cc1b493eaff P 36dde0a7a9114639bc27dca417b288cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 36dde0a7a9114639bc27dca417b288cc (127.0.97.1). New cstate: current_term: 1 leader_uuid: "36dde0a7a9114639bc27dca417b288cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dce3d411234778a20049dc1d077dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f77615a1c5e4961862b33a057d44c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36dde0a7a9114639bc27dca417b288cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46659 } health_report { overall_health: HEALTHY } } }
I20260929 15:51:28.286134 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:28.279968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42210 (local address 127.0.97.1:46659)
0929 15:51:28.280213 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:28.280219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:28.280333 (+ 114us) server_negotiation.cc:407] Connection header received
0929 15:51:28.280691 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:28.280696 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:28.280781 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:28.280945 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:51:28.281314 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:28.282504 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:28.284013 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:28.284398 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:28.284492 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:28.284973 (+ 481us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:51:28.285544 (+ 571us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:51:28.285678 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:28.285787 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 15:51:28.285927 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 15:51:28.293625 1811 raft_consensus.cc:1261] T dcfb01a790f240708dfc9cc1b493eaff P 8f77615a1c5e4961862b33a057d44c3c [term 1 FOLLOWER]: Refusing update from remote peer 36dde0a7a9114639bc27dca417b288cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:51:28.293663 1950 raft_consensus.cc:1261] T dcfb01a790f240708dfc9cc1b493eaff P f9dce3d411234778a20049dc1d077dd2 [term 1 FOLLOWER]: Refusing update from remote peer 36dde0a7a9114639bc27dca417b288cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:51:28.294564 2025 consensus_queue.cc:1035] T dcfb01a790f240708dfc9cc1b493eaff P 36dde0a7a9114639bc27dca417b288cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9dce3d411234778a20049dc1d077dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:51:28.294889 2022 consensus_queue.cc:1035] T dcfb01a790f240708dfc9cc1b493eaff P 36dde0a7a9114639bc27dca417b288cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f77615a1c5e4961862b33a057d44c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:51:28.313275 2028 mvcc.cc:205] Tried to move back new op lower bound from 7334695273641799680 to 7334695273287876608. Current Snapshot: MvccSnapshot[applied={T|T < 7334695273641799680 or (T in {7334695273641799680})}]
I20260929 15:51:28.316905 2030 mvcc.cc:205] Tried to move back new op lower bound from 7334695273641799680 to 7334695273287876608. Current Snapshot: MvccSnapshot[applied={T|T < 7334695273641799680}]
I20260929 15:51:28.336458 1510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:51:28.336776 1510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:51:28.340677 1510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6f7e0dcaa3c4ecd9658ee15fb3594ad: Sending DeleteTablet for 3 replicas of tablet dcfb01a790f240708dfc9cc1b493eaff
I20260929 15:51:28.342090 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet dcfb01a790f240708dfc9cc1b493eaff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:51:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42208
I20260929 15:51:28.342191 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet dcfb01a790f240708dfc9cc1b493eaff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:51:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40016
I20260929 15:51:28.343356 1786 tablet_service.cc:1514] Processing DeleteTablet for tablet dcfb01a790f240708dfc9cc1b493eaff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:51:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34156
I20260929 15:51:28.351280 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43342:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:51:28.351950 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:51:28.360755 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 6610c21fbef04604bd0013f5a0cced2a (DEFAULT_TABLE table=test-table [id=069ce536d1124596bfc5a9f7e922cb7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:51:28.361280 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6610c21fbef04604bd0013f5a0cced2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:28.366389 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 6610c21fbef04604bd0013f5a0cced2a (DEFAULT_TABLE table=test-table [id=069ce536d1124596bfc5a9f7e922cb7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:51:28.367041 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6610c21fbef04604bd0013f5a0cced2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:28.370533 2042 tablet_replica.cc:330] T dcfb01a790f240708dfc9cc1b493eaff P 8f77615a1c5e4961862b33a057d44c3c: stopping tablet replica
I20260929 15:51:28.371140 2042 raft_consensus.cc:2229] T dcfb01a790f240708dfc9cc1b493eaff P 8f77615a1c5e4961862b33a057d44c3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:51:28.371675 2042 raft_consensus.cc:2258] T dcfb01a790f240708dfc9cc1b493eaff P 8f77615a1c5e4961862b33a057d44c3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:51:28.372540 2040 tablet_replica.cc:330] T dcfb01a790f240708dfc9cc1b493eaff P 36dde0a7a9114639bc27dca417b288cc: stopping tablet replica
I20260929 15:51:28.372980 2040 raft_consensus.cc:2229] T dcfb01a790f240708dfc9cc1b493eaff P 36dde0a7a9114639bc27dca417b288cc [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:51:28.374660 1646 tablet_service.cc:1467] Processing CreateTablet for tablet 6610c21fbef04604bd0013f5a0cced2a (DEFAULT_TABLE table=test-table [id=069ce536d1124596bfc5a9f7e922cb7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:51:28.375226 1646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6610c21fbef04604bd0013f5a0cced2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:28.375810 2042 ts_tablet_manager.cc:1905] T dcfb01a790f240708dfc9cc1b493eaff P 8f77615a1c5e4961862b33a057d44c3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:51:28.376333 2040 raft_consensus.cc:2258] T dcfb01a790f240708dfc9cc1b493eaff P 36dde0a7a9114639bc27dca417b288cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:51:28.377101 2041 tablet_replica.cc:330] T dcfb01a790f240708dfc9cc1b493eaff P f9dce3d411234778a20049dc1d077dd2: stopping tablet replica
I20260929 15:51:28.377584 2041 raft_consensus.cc:2229] T dcfb01a790f240708dfc9cc1b493eaff P f9dce3d411234778a20049dc1d077dd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:51:28.378037 2041 raft_consensus.cc:2258] T dcfb01a790f240708dfc9cc1b493eaff P f9dce3d411234778a20049dc1d077dd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:51:28.379740 2040 ts_tablet_manager.cc:1905] T dcfb01a790f240708dfc9cc1b493eaff P 36dde0a7a9114639bc27dca417b288cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:51:28.381803 2015 tablet_bootstrap.cc:492] T 6610c21fbef04604bd0013f5a0cced2a P 36dde0a7a9114639bc27dca417b288cc: Bootstrap starting.
I20260929 15:51:28.382488 2014 tablet_bootstrap.cc:492] T 6610c21fbef04604bd0013f5a0cced2a P 8f77615a1c5e4961862b33a057d44c3c: Bootstrap starting.
I20260929 15:51:28.382591 2016 tablet_bootstrap.cc:492] T 6610c21fbef04604bd0013f5a0cced2a P f9dce3d411234778a20049dc1d077dd2: Bootstrap starting.
I20260929 15:51:28.383407 2015 tablet_bootstrap.cc:654] T 6610c21fbef04604bd0013f5a0cced2a P 36dde0a7a9114639bc27dca417b288cc: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:28.383951 2042 ts_tablet_manager.cc:1918] T dcfb01a790f240708dfc9cc1b493eaff P 8f77615a1c5e4961862b33a057d44c3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:51:28.384073 2014 tablet_bootstrap.cc:654] T 6610c21fbef04604bd0013f5a0cced2a P 8f77615a1c5e4961862b33a057d44c3c: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:28.384251 2042 log.cc:1196] T dcfb01a790f240708dfc9cc1b493eaff P 8f77615a1c5e4961862b33a057d44c3c: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-1/wal/wals/dcfb01a790f240708dfc9cc1b493eaff
I20260929 15:51:28.384558 2016 tablet_bootstrap.cc:654] T 6610c21fbef04604bd0013f5a0cced2a P f9dce3d411234778a20049dc1d077dd2: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:28.385010 2042 ts_tablet_manager.cc:1939] T dcfb01a790f240708dfc9cc1b493eaff P 8f77615a1c5e4961862b33a057d44c3c: Deleting consensus metadata
I20260929 15:51:28.387217 1497 catalog_manager.cc:4841] TS 8f77615a1c5e4961862b33a057d44c3c (127.0.97.2:34683): tablet dcfb01a790f240708dfc9cc1b493eaff (table test-table [id=0631e37203dd46dfae1270bb821558f0]) successfully deleted
I20260929 15:51:28.390444 2041 ts_tablet_manager.cc:1905] T dcfb01a790f240708dfc9cc1b493eaff P f9dce3d411234778a20049dc1d077dd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:51:28.396332 2040 ts_tablet_manager.cc:1918] T dcfb01a790f240708dfc9cc1b493eaff P 36dde0a7a9114639bc27dca417b288cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:51:28.396584 2040 log.cc:1196] T dcfb01a790f240708dfc9cc1b493eaff P 36dde0a7a9114639bc27dca417b288cc: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-0/wal/wals/dcfb01a790f240708dfc9cc1b493eaff
I20260929 15:51:28.397163 2040 ts_tablet_manager.cc:1939] T dcfb01a790f240708dfc9cc1b493eaff P 36dde0a7a9114639bc27dca417b288cc: Deleting consensus metadata
I20260929 15:51:28.402158 1498 catalog_manager.cc:4841] TS 36dde0a7a9114639bc27dca417b288cc (127.0.97.1:46659): tablet dcfb01a790f240708dfc9cc1b493eaff (table test-table [id=0631e37203dd46dfae1270bb821558f0]) successfully deleted
I20260929 15:51:28.406468 2041 ts_tablet_manager.cc:1918] T dcfb01a790f240708dfc9cc1b493eaff P f9dce3d411234778a20049dc1d077dd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:51:28.406618 2041 log.cc:1196] T dcfb01a790f240708dfc9cc1b493eaff P f9dce3d411234778a20049dc1d077dd2: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-2/wal/wals/dcfb01a790f240708dfc9cc1b493eaff
I20260929 15:51:28.407100 2041 ts_tablet_manager.cc:1939] T dcfb01a790f240708dfc9cc1b493eaff P f9dce3d411234778a20049dc1d077dd2: Deleting consensus metadata
I20260929 15:51:28.408746 1496 catalog_manager.cc:4841] TS f9dce3d411234778a20049dc1d077dd2 (127.0.97.3:42325): tablet dcfb01a790f240708dfc9cc1b493eaff (table test-table [id=0631e37203dd46dfae1270bb821558f0]) successfully deleted
I20260929 15:51:28.420218 2015 tablet_bootstrap.cc:492] T 6610c21fbef04604bd0013f5a0cced2a P 36dde0a7a9114639bc27dca417b288cc: No bootstrap required, opened a new log
I20260929 15:51:28.420414 2015 ts_tablet_manager.cc:1397] T 6610c21fbef04604bd0013f5a0cced2a P 36dde0a7a9114639bc27dca417b288cc: Time spent bootstrapping tablet: real 0.039s user 0.000s sys 0.010s
I20260929 15:51:28.421468 2015 raft_consensus.cc:348] T 6610c21fbef04604bd0013f5a0cced2a P 36dde0a7a9114639bc27dca417b288cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36dde0a7a9114639bc27dca417b288cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46659 } } peers { permanent_uuid: "8f77615a1c5e4961862b33a057d44c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34683 } } peers { permanent_uuid: "f9dce3d411234778a20049dc1d077dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42325 } }
I20260929 15:51:28.421765 2015 raft_consensus.cc:374] T 6610c21fbef04604bd0013f5a0cced2a P 36dde0a7a9114639bc27dca417b288cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:28.421847 2015 raft_consensus.cc:729] T 6610c21fbef04604bd0013f5a0cced2a P 36dde0a7a9114639bc27dca417b288cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36dde0a7a9114639bc27dca417b288cc, State: Initialized, Role: FOLLOWER
I20260929 15:51:28.422185 2015 consensus_queue.cc:260] T 6610c21fbef04604bd0013f5a0cced2a P 36dde0a7a9114639bc27dca417b288cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36dde0a7a9114639bc27dca417b288cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46659 } } peers { permanent_uuid: "8f77615a1c5e4961862b33a057d44c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34683 } } peers { permanent_uuid: "f9dce3d411234778a20049dc1d077dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42325 } }
I20260929 15:51:28.423019 2014 tablet_bootstrap.cc:492] T 6610c21fbef04604bd0013f5a0cced2a P 8f77615a1c5e4961862b33a057d44c3c: No bootstrap required, opened a new log
I20260929 15:51:28.423177 2014 ts_tablet_manager.cc:1397] T 6610c21fbef04604bd0013f5a0cced2a P 8f77615a1c5e4961862b33a057d44c3c: Time spent bootstrapping tablet: real 0.041s user 0.007s sys 0.027s
I20260929 15:51:28.423992 2014 raft_consensus.cc:348] T 6610c21fbef04604bd0013f5a0cced2a P 8f77615a1c5e4961862b33a057d44c3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36dde0a7a9114639bc27dca417b288cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46659 } } peers { permanent_uuid: "8f77615a1c5e4961862b33a057d44c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34683 } } peers { permanent_uuid: "f9dce3d411234778a20049dc1d077dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42325 } }
I20260929 15:51:28.424243 2014 raft_consensus.cc:374] T 6610c21fbef04604bd0013f5a0cced2a P 8f77615a1c5e4961862b33a057d44c3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:28.424337 2014 raft_consensus.cc:729] T 6610c21fbef04604bd0013f5a0cced2a P 8f77615a1c5e4961862b33a057d44c3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f77615a1c5e4961862b33a057d44c3c, State: Initialized, Role: FOLLOWER
I20260929 15:51:28.424796 2015 ts_tablet_manager.cc:1428] T 6610c21fbef04604bd0013f5a0cced2a P 36dde0a7a9114639bc27dca417b288cc: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260929 15:51:28.425482 2016 tablet_bootstrap.cc:492] T 6610c21fbef04604bd0013f5a0cced2a P f9dce3d411234778a20049dc1d077dd2: No bootstrap required, opened a new log
I20260929 15:51:28.425629 2016 ts_tablet_manager.cc:1397] T 6610c21fbef04604bd0013f5a0cced2a P f9dce3d411234778a20049dc1d077dd2: Time spent bootstrapping tablet: real 0.043s user 0.017s sys 0.011s
I20260929 15:51:28.426357 2016 raft_consensus.cc:348] T 6610c21fbef04604bd0013f5a0cced2a P f9dce3d411234778a20049dc1d077dd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36dde0a7a9114639bc27dca417b288cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46659 } } peers { permanent_uuid: "8f77615a1c5e4961862b33a057d44c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34683 } } peers { permanent_uuid: "f9dce3d411234778a20049dc1d077dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42325 } }
I20260929 15:51:28.426584 2016 raft_consensus.cc:374] T 6610c21fbef04604bd0013f5a0cced2a P f9dce3d411234778a20049dc1d077dd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:28.426678 2016 raft_consensus.cc:729] T 6610c21fbef04604bd0013f5a0cced2a P f9dce3d411234778a20049dc1d077dd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9dce3d411234778a20049dc1d077dd2, State: Initialized, Role: FOLLOWER
I20260929 15:51:28.426827 2014 consensus_queue.cc:260] T 6610c21fbef04604bd0013f5a0cced2a P 8f77615a1c5e4961862b33a057d44c3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36dde0a7a9114639bc27dca417b288cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46659 } } peers { permanent_uuid: "8f77615a1c5e4961862b33a057d44c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34683 } } peers { permanent_uuid: "f9dce3d411234778a20049dc1d077dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42325 } }
I20260929 15:51:28.426964 2016 consensus_queue.cc:260] T 6610c21fbef04604bd0013f5a0cced2a P f9dce3d411234778a20049dc1d077dd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36dde0a7a9114639bc27dca417b288cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46659 } } peers { permanent_uuid: "8f77615a1c5e4961862b33a057d44c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34683 } } peers { permanent_uuid: "f9dce3d411234778a20049dc1d077dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42325 } }
I20260929 15:51:28.427947 2016 ts_tablet_manager.cc:1428] T 6610c21fbef04604bd0013f5a0cced2a P f9dce3d411234778a20049dc1d077dd2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 15:51:28.432567 2014 ts_tablet_manager.cc:1428] T 6610c21fbef04604bd0013f5a0cced2a P 8f77615a1c5e4961862b33a057d44c3c: Time spent starting tablet: real 0.009s user 0.001s sys 0.002s
I20260929 15:51:28.672415 2020 raft_consensus.cc:482] T 6610c21fbef04604bd0013f5a0cced2a P f9dce3d411234778a20049dc1d077dd2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:51:28.672665 2020 raft_consensus.cc:504] T 6610c21fbef04604bd0013f5a0cced2a P f9dce3d411234778a20049dc1d077dd2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36dde0a7a9114639bc27dca417b288cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46659 } } peers { permanent_uuid: "8f77615a1c5e4961862b33a057d44c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34683 } } peers { permanent_uuid: "f9dce3d411234778a20049dc1d077dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42325 } }
I20260929 15:51:28.674299 2020 leader_election.cc:288] T 6610c21fbef04604bd0013f5a0cced2a P f9dce3d411234778a20049dc1d077dd2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36dde0a7a9114639bc27dca417b288cc (127.0.97.1:46659), 8f77615a1c5e4961862b33a057d44c3c (127.0.97.2:34683)
I20260929 15:51:28.682493 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:28.674956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51785 (local address 127.0.97.2:34683)
0929 15:51:28.675834 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:28.675840 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:28.675871 (+ 31us) server_negotiation.cc:407] Connection header received
0929 15:51:28.675944 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:28.675950 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:28.676023 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:28.676143 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:51:28.676664 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:28.677817 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:28.680176 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:28.681365 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:28.681460 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:28.681555 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:28.682049 (+ 494us) server_negotiation.cc:299] Negotiation successful
0929 15:51:28.682185 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":745}
I20260929 15:51:28.683204 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:28.674584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34683 (local address 127.0.97.3:51785)
0929 15:51:28.675505 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:28.675518 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:28.675590 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:28.676156 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:28.676160 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:28.676182 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:51:28.676458 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:28.676472 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:28.678009 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:28.678017 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:28.680012 (+ 1995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:28.680024 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:28.681688 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:28.681693 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:28.681874 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:28.681905 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 15:51:28.682734 (+ 829us) client_negotiation.cc:240] Negotiation successful
0929 15:51:28.682877 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"thread_start_us":428,"threads_started":1}
I20260929 15:51:28.683843 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6610c21fbef04604bd0013f5a0cced2a" candidate_uuid: "f9dce3d411234778a20049dc1d077dd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f77615a1c5e4961862b33a057d44c3c" is_pre_election: true
I20260929 15:51:28.684105 1811 raft_consensus.cc:2454] T 6610c21fbef04604bd0013f5a0cced2a P 8f77615a1c5e4961862b33a057d44c3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9dce3d411234778a20049dc1d077dd2 in term 0.
I20260929 15:51:28.684644 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:28.676380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36383 (local address 127.0.97.1:46659)
0929 15:51:28.676610 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:28.676618 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:28.676636 (+ 18us) server_negotiation.cc:407] Connection header received
0929 15:51:28.676701 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:28.676711 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:28.676786 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:28.676885 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:51:28.677769 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:28.678951 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:28.681268 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:28.682558 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:28.682673 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:28.682775 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:28.684187 (+ 1412us) server_negotiation.cc:299] Negotiation successful
0929 15:51:28.684324 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 15:51:28.684633 1878 leader_election.cc:302] T 6610c21fbef04604bd0013f5a0cced2a P f9dce3d411234778a20049dc1d077dd2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f77615a1c5e4961862b33a057d44c3c, f9dce3d411234778a20049dc1d077dd2; no voters:
I20260929 15:51:28.685055 2020 raft_consensus.cc:2781] T 6610c21fbef04604bd0013f5a0cced2a P f9dce3d411234778a20049dc1d077dd2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:51:28.685168 2020 raft_consensus.cc:482] T 6610c21fbef04604bd0013f5a0cced2a P f9dce3d411234778a20049dc1d077dd2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:51:28.685262 2020 raft_consensus.cc:3037] T 6610c21fbef04604bd0013f5a0cced2a P f9dce3d411234778a20049dc1d077dd2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:28.685724 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:28.674571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46659 (local address 127.0.97.3:36383)
0929 15:51:28.675068 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:28.675092 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:28.675235 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:28.677223 (+ 1988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:28.677229 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:28.677252 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:51:28.677578 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:28.677588 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:28.679130 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:28.679135 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:28.681099 (+ 1964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:28.681114 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:28.683165 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:28.683172 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:28.683937 (+ 765us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:28.683968 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 15:51:28.685017 (+ 1049us) client_negotiation.cc:240] Negotiation successful
0929 15:51:28.685215 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":1,"thread_start_us":2473,"threads_started":1}
I20260929 15:51:28.686822 1672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6610c21fbef04604bd0013f5a0cced2a" candidate_uuid: "f9dce3d411234778a20049dc1d077dd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36dde0a7a9114639bc27dca417b288cc" is_pre_election: true
I20260929 15:51:28.687282 1672 raft_consensus.cc:2454] T 6610c21fbef04604bd0013f5a0cced2a P 36dde0a7a9114639bc27dca417b288cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9dce3d411234778a20049dc1d077dd2 in term 0.
I20260929 15:51:28.688488 2020 raft_consensus.cc:504] T 6610c21fbef04604bd0013f5a0cced2a P f9dce3d411234778a20049dc1d077dd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36dde0a7a9114639bc27dca417b288cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46659 } } peers { permanent_uuid: "8f77615a1c5e4961862b33a057d44c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34683 } } peers { permanent_uuid: "f9dce3d411234778a20049dc1d077dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42325 } }
I20260929 15:51:28.689507 1672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6610c21fbef04604bd0013f5a0cced2a" candidate_uuid: "f9dce3d411234778a20049dc1d077dd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36dde0a7a9114639bc27dca417b288cc"
I20260929 15:51:28.689689 1672 raft_consensus.cc:3037] T 6610c21fbef04604bd0013f5a0cced2a P 36dde0a7a9114639bc27dca417b288cc [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:28.691229 2020 leader_election.cc:288] T 6610c21fbef04604bd0013f5a0cced2a P f9dce3d411234778a20049dc1d077dd2 [CANDIDATE]: Term 1 election: Requested vote from peers 36dde0a7a9114639bc27dca417b288cc (127.0.97.1:46659), 8f77615a1c5e4961862b33a057d44c3c (127.0.97.2:34683)
I20260929 15:51:28.691969 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6610c21fbef04604bd0013f5a0cced2a" candidate_uuid: "f9dce3d411234778a20049dc1d077dd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f77615a1c5e4961862b33a057d44c3c"
I20260929 15:51:28.692169 1811 raft_consensus.cc:3037] T 6610c21fbef04604bd0013f5a0cced2a P 8f77615a1c5e4961862b33a057d44c3c [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:28.692762 1672 raft_consensus.cc:2454] T 6610c21fbef04604bd0013f5a0cced2a P 36dde0a7a9114639bc27dca417b288cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9dce3d411234778a20049dc1d077dd2 in term 1.
I20260929 15:51:28.693148 1879 leader_election.cc:302] T 6610c21fbef04604bd0013f5a0cced2a P f9dce3d411234778a20049dc1d077dd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36dde0a7a9114639bc27dca417b288cc, f9dce3d411234778a20049dc1d077dd2; no voters:
I20260929 15:51:28.693470 2020 raft_consensus.cc:2781] T 6610c21fbef04604bd0013f5a0cced2a P f9dce3d411234778a20049dc1d077dd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:51:28.695261 1811 raft_consensus.cc:2454] T 6610c21fbef04604bd0013f5a0cced2a P 8f77615a1c5e4961862b33a057d44c3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9dce3d411234778a20049dc1d077dd2 in term 1.
I20260929 15:51:28.696023 2020 raft_consensus.cc:686] T 6610c21fbef04604bd0013f5a0cced2a P f9dce3d411234778a20049dc1d077dd2 [term 1 LEADER]: Becoming Leader. State: Replica: f9dce3d411234778a20049dc1d077dd2, State: Running, Role: LEADER
I20260929 15:51:28.696442 2020 consensus_queue.cc:237] T 6610c21fbef04604bd0013f5a0cced2a P f9dce3d411234778a20049dc1d077dd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36dde0a7a9114639bc27dca417b288cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46659 } } peers { permanent_uuid: "8f77615a1c5e4961862b33a057d44c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34683 } } peers { permanent_uuid: "f9dce3d411234778a20049dc1d077dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42325 } }
I20260929 15:51:28.700405 1510 catalog_manager.cc:5495] T 6610c21fbef04604bd0013f5a0cced2a P f9dce3d411234778a20049dc1d077dd2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9dce3d411234778a20049dc1d077dd2 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "f9dce3d411234778a20049dc1d077dd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36dde0a7a9114639bc27dca417b288cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f77615a1c5e4961862b33a057d44c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9dce3d411234778a20049dc1d077dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42325 } health_report { overall_health: HEALTHY } } }
Sep 29 15:51:28 dist-test-slave-r0ck krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:51:28.812271 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:28.791440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33219 (local address 127.0.0.1:43350)
0929 15:51:28.792895 (+ 1455us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:28.792922 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:28.793087 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:28.793420 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:28.793425 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:28.794065 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:51:28.794232 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:28.794241 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:28.795256 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:28.795262 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:28.796175 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:28.796186 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:28.796879 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:28.796892 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:28.797081 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:28.801023 (+ 3942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:51:28.801070 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:51:28.804595 (+ 3525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:51:28.809127 (+ 4532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:28.809142 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:28.809149 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:28.809841 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:28.810250 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:28.810253 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:28.810256 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:28.810403 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:28.811146 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:51:28.811155 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:51:28.811392 (+ 237us) client_negotiation.cc:769] Sending connection context
0929 15:51:28.811586 (+ 194us) client_negotiation.cc:240] Negotiation successful
0929 15:51:28.811852 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1045,"thread_start_us":900,"threads_started":1}
I20260929 15:51:28.813100 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:28.791440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43350 (local address 127.0.97.62:33219)
0929 15:51:28.792864 (+ 1424us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:28.792870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:28.793007 (+ 137us) server_negotiation.cc:407] Connection header received
0929 15:51:28.793216 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:28.793219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:28.793270 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:28.793395 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:28.794383 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:28.795120 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:28.796308 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:28.796721 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:28.796865 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:28.804840 (+ 7975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:28.804847 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:28.804853 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:28.804911 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:28.808553 (+ 3642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:28.809980 (+ 1427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:28.809985 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:28.809989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:28.810133 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:28.810526 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:28.810534 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:28.810541 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:28.810999 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:28.811157 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:28.812578 (+ 1421us) server_negotiation.cc:299] Negotiation successful
0929 15:51:28.812732 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":897,"thread_start_us":313,"threads_started":1}
I20260929 15:51:28.819568 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43350:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 15:51:28.820812 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:51:28.828750 1924 tablet_service.cc:1467] Processing CreateTablet for tablet c9aa43ee18124167bf85018fb2bf1e49 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c60c25296807470dad41f08773934af8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 15:51:28.828750 1646 tablet_service.cc:1467] Processing CreateTablet for tablet c9aa43ee18124167bf85018fb2bf1e49 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c60c25296807470dad41f08773934af8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 15:51:28.828991 1785 tablet_service.cc:1467] Processing CreateTablet for tablet c9aa43ee18124167bf85018fb2bf1e49 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c60c25296807470dad41f08773934af8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 15:51:28.829308 1646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9aa43ee18124167bf85018fb2bf1e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:28.829300 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9aa43ee18124167bf85018fb2bf1e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:28.829562 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9aa43ee18124167bf85018fb2bf1e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:28.834476 2014 tablet_bootstrap.cc:492] T c9aa43ee18124167bf85018fb2bf1e49 P 8f77615a1c5e4961862b33a057d44c3c: Bootstrap starting.
I20260929 15:51:28.836377 2016 tablet_bootstrap.cc:492] T c9aa43ee18124167bf85018fb2bf1e49 P f9dce3d411234778a20049dc1d077dd2: Bootstrap starting.
I20260929 15:51:28.836549 2014 tablet_bootstrap.cc:654] T c9aa43ee18124167bf85018fb2bf1e49 P 8f77615a1c5e4961862b33a057d44c3c: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:28.837949 2016 tablet_bootstrap.cc:654] T c9aa43ee18124167bf85018fb2bf1e49 P f9dce3d411234778a20049dc1d077dd2: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:28.839768 2015 tablet_bootstrap.cc:492] T c9aa43ee18124167bf85018fb2bf1e49 P 36dde0a7a9114639bc27dca417b288cc: Bootstrap starting.
I20260929 15:51:28.840976 2016 tablet_bootstrap.cc:492] T c9aa43ee18124167bf85018fb2bf1e49 P f9dce3d411234778a20049dc1d077dd2: No bootstrap required, opened a new log
I20260929 15:51:28.841012 2014 tablet_bootstrap.cc:492] T c9aa43ee18124167bf85018fb2bf1e49 P 8f77615a1c5e4961862b33a057d44c3c: No bootstrap required, opened a new log
I20260929 15:51:28.841132 2016 ts_tablet_manager.cc:1397] T c9aa43ee18124167bf85018fb2bf1e49 P f9dce3d411234778a20049dc1d077dd2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 15:51:28.841135 2014 ts_tablet_manager.cc:1397] T c9aa43ee18124167bf85018fb2bf1e49 P 8f77615a1c5e4961862b33a057d44c3c: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260929 15:51:28.841336 2015 tablet_bootstrap.cc:654] T c9aa43ee18124167bf85018fb2bf1e49 P 36dde0a7a9114639bc27dca417b288cc: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:28.841837 2016 raft_consensus.cc:348] T c9aa43ee18124167bf85018fb2bf1e49 P f9dce3d411234778a20049dc1d077dd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36dde0a7a9114639bc27dca417b288cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46659 } } peers { permanent_uuid: "8f77615a1c5e4961862b33a057d44c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34683 } } peers { permanent_uuid: "f9dce3d411234778a20049dc1d077dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42325 } }
I20260929 15:51:28.841835 2014 raft_consensus.cc:348] T c9aa43ee18124167bf85018fb2bf1e49 P 8f77615a1c5e4961862b33a057d44c3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36dde0a7a9114639bc27dca417b288cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46659 } } peers { permanent_uuid: "8f77615a1c5e4961862b33a057d44c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34683 } } peers { permanent_uuid: "f9dce3d411234778a20049dc1d077dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42325 } }
I20260929 15:51:28.842067 2014 raft_consensus.cc:374] T c9aa43ee18124167bf85018fb2bf1e49 P 8f77615a1c5e4961862b33a057d44c3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:28.842068 2016 raft_consensus.cc:374] T c9aa43ee18124167bf85018fb2bf1e49 P f9dce3d411234778a20049dc1d077dd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:28.842139 2014 raft_consensus.cc:729] T c9aa43ee18124167bf85018fb2bf1e49 P 8f77615a1c5e4961862b33a057d44c3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f77615a1c5e4961862b33a057d44c3c, State: Initialized, Role: FOLLOWER
I20260929 15:51:28.842139 2016 raft_consensus.cc:729] T c9aa43ee18124167bf85018fb2bf1e49 P f9dce3d411234778a20049dc1d077dd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9dce3d411234778a20049dc1d077dd2, State: Initialized, Role: FOLLOWER
I20260929 15:51:28.842399 2016 consensus_queue.cc:260] T c9aa43ee18124167bf85018fb2bf1e49 P f9dce3d411234778a20049dc1d077dd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36dde0a7a9114639bc27dca417b288cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46659 } } peers { permanent_uuid: "8f77615a1c5e4961862b33a057d44c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34683 } } peers { permanent_uuid: "f9dce3d411234778a20049dc1d077dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42325 } }
I20260929 15:51:28.842387 2014 consensus_queue.cc:260] T c9aa43ee18124167bf85018fb2bf1e49 P 8f77615a1c5e4961862b33a057d44c3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36dde0a7a9114639bc27dca417b288cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46659 } } peers { permanent_uuid: "8f77615a1c5e4961862b33a057d44c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34683 } } peers { permanent_uuid: "f9dce3d411234778a20049dc1d077dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42325 } }
I20260929 15:51:28.842774 2020 tablet_replica.cc:439] T c9aa43ee18124167bf85018fb2bf1e49 P f9dce3d411234778a20049dc1d077dd2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36dde0a7a9114639bc27dca417b288cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46659 } } peers { permanent_uuid: "8f77615a1c5e4961862b33a057d44c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34683 } } peers { permanent_uuid: "f9dce3d411234778a20049dc1d077dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42325 } } }
I20260929 15:51:28.842927 2014 ts_tablet_manager.cc:1428] T c9aa43ee18124167bf85018fb2bf1e49 P 8f77615a1c5e4961862b33a057d44c3c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 15:51:28.843041 2020 tablet_replica.cc:442] T c9aa43ee18124167bf85018fb2bf1e49 P f9dce3d411234778a20049dc1d077dd2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:51:28.843216 2021 tablet_replica.cc:439] T c9aa43ee18124167bf85018fb2bf1e49 P 8f77615a1c5e4961862b33a057d44c3c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36dde0a7a9114639bc27dca417b288cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46659 } } peers { permanent_uuid: "8f77615a1c5e4961862b33a057d44c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34683 } } peers { permanent_uuid: "f9dce3d411234778a20049dc1d077dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42325 } } }
I20260929 15:51:28.843530 2021 tablet_replica.cc:442] T c9aa43ee18124167bf85018fb2bf1e49 P 8f77615a1c5e4961862b33a057d44c3c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:51:28.843931 2016 ts_tablet_manager.cc:1428] T c9aa43ee18124167bf85018fb2bf1e49 P f9dce3d411234778a20049dc1d077dd2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 15:51:28.845058 2015 tablet_bootstrap.cc:492] T c9aa43ee18124167bf85018fb2bf1e49 P 36dde0a7a9114639bc27dca417b288cc: No bootstrap required, opened a new log
I20260929 15:51:28.845175 2015 ts_tablet_manager.cc:1397] T c9aa43ee18124167bf85018fb2bf1e49 P 36dde0a7a9114639bc27dca417b288cc: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 15:51:28.845840 2015 raft_consensus.cc:348] T c9aa43ee18124167bf85018fb2bf1e49 P 36dde0a7a9114639bc27dca417b288cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36dde0a7a9114639bc27dca417b288cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46659 } } peers { permanent_uuid: "8f77615a1c5e4961862b33a057d44c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34683 } } peers { permanent_uuid: "f9dce3d411234778a20049dc1d077dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42325 } }
I20260929 15:51:28.846071 2015 raft_consensus.cc:374] T c9aa43ee18124167bf85018fb2bf1e49 P 36dde0a7a9114639bc27dca417b288cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:28.846155 2015 raft_consensus.cc:729] T c9aa43ee18124167bf85018fb2bf1e49 P 36dde0a7a9114639bc27dca417b288cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36dde0a7a9114639bc27dca417b288cc, State: Initialized, Role: FOLLOWER
I20260929 15:51:28.846381 2015 consensus_queue.cc:260] T c9aa43ee18124167bf85018fb2bf1e49 P 36dde0a7a9114639bc27dca417b288cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36dde0a7a9114639bc27dca417b288cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46659 } } peers { permanent_uuid: "8f77615a1c5e4961862b33a057d44c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34683 } } peers { permanent_uuid: "f9dce3d411234778a20049dc1d077dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42325 } }
I20260929 15:51:28.846791 2025 tablet_replica.cc:439] T c9aa43ee18124167bf85018fb2bf1e49 P 36dde0a7a9114639bc27dca417b288cc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36dde0a7a9114639bc27dca417b288cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46659 } } peers { permanent_uuid: "8f77615a1c5e4961862b33a057d44c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34683 } } peers { permanent_uuid: "f9dce3d411234778a20049dc1d077dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42325 } } }
I20260929 15:51:28.846974 2015 ts_tablet_manager.cc:1428] T c9aa43ee18124167bf85018fb2bf1e49 P 36dde0a7a9114639bc27dca417b288cc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 15:51:28.847067 2025 tablet_replica.cc:442] T c9aa43ee18124167bf85018fb2bf1e49 P 36dde0a7a9114639bc27dca417b288cc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:51:28.972983 2021 raft_consensus.cc:482] T c9aa43ee18124167bf85018fb2bf1e49 P 8f77615a1c5e4961862b33a057d44c3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:51:28.973194 2021 raft_consensus.cc:504] T c9aa43ee18124167bf85018fb2bf1e49 P 8f77615a1c5e4961862b33a057d44c3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36dde0a7a9114639bc27dca417b288cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46659 } } peers { permanent_uuid: "8f77615a1c5e4961862b33a057d44c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34683 } } peers { permanent_uuid: "f9dce3d411234778a20049dc1d077dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42325 } }
I20260929 15:51:28.974735 2021 leader_election.cc:288] T c9aa43ee18124167bf85018fb2bf1e49 P 8f77615a1c5e4961862b33a057d44c3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36dde0a7a9114639bc27dca417b288cc (127.0.97.1:46659), f9dce3d411234778a20049dc1d077dd2 (127.0.97.3:42325)
I20260929 15:51:28.981290 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:28.975084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51675 (local address 127.0.97.1:46659)
0929 15:51:28.975282 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:28.975292 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:28.976226 (+ 934us) server_negotiation.cc:407] Connection header received
0929 15:51:28.976402 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:28.976410 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:28.976502 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:28.976610 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:51:28.977132 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:28.977952 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:28.979645 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:28.980398 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:28.980550 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:28.980642 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:28.980895 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 15:51:28.981030 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 15:51:28.981431 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:28.975110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46659 (local address 127.0.97.2:51675)
0929 15:51:28.976132 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:28.976145 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:28.976263 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:28.976667 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:28.976670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:28.976701 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:51:28.976940 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:28.976948 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:28.978085 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:28.978090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:28.979490 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:28.979501 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:28.980576 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:28.980582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:28.980740 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:28.980763 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 15:51:28.980872 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 15:51:28.981000 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":933,"mutex_wait_us":44,"thread_start_us":807,"threads_started":1}
I20260929 15:51:28.981685 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:28.975130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46223 (local address 127.0.97.3:42325)
0929 15:51:28.976704 (+ 1574us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:28.976710 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:28.976729 (+ 19us) server_negotiation.cc:407] Connection header received
0929 15:51:28.976792 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:28.976795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:28.976855 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:28.976966 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:51:28.977533 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:28.978294 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:28.979962 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:28.980705 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:28.980857 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:28.980956 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:28.981233 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 15:51:28.981409 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1442,"thread_start_us":1334,"threads_started":1}
I20260929 15:51:28.981743 2060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:28.975068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42325 (local address 127.0.97.2:46223)
0929 15:51:28.975656 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:28.975677 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:28.975775 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:28.976985 (+ 1210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:28.976990 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:28.977016 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:51:28.977381 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:28.977390 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:28.978427 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:28.978437 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:28.979809 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:28.979819 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:28.980923 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:28.980930 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:28.981081 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:28.981102 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 15:51:28.981211 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 15:51:28.981330 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":1,"thread_start_us":393,"threads_started":1}
I20260929 15:51:28.982343 1672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9aa43ee18124167bf85018fb2bf1e49" candidate_uuid: "8f77615a1c5e4961862b33a057d44c3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36dde0a7a9114639bc27dca417b288cc" is_pre_election: true
I20260929 15:51:28.982371 1950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9aa43ee18124167bf85018fb2bf1e49" candidate_uuid: "8f77615a1c5e4961862b33a057d44c3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9dce3d411234778a20049dc1d077dd2" is_pre_election: true
I20260929 15:51:28.982595 1672 raft_consensus.cc:2454] T c9aa43ee18124167bf85018fb2bf1e49 P 36dde0a7a9114639bc27dca417b288cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f77615a1c5e4961862b33a057d44c3c in term 0.
I20260929 15:51:28.982602 1950 raft_consensus.cc:2454] T c9aa43ee18124167bf85018fb2bf1e49 P f9dce3d411234778a20049dc1d077dd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f77615a1c5e4961862b33a057d44c3c in term 0.
I20260929 15:51:28.983070 1740 leader_election.cc:302] T c9aa43ee18124167bf85018fb2bf1e49 P 8f77615a1c5e4961862b33a057d44c3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 36dde0a7a9114639bc27dca417b288cc, 8f77615a1c5e4961862b33a057d44c3c, f9dce3d411234778a20049dc1d077dd2; no voters:
I20260929 15:51:28.983495 2021 raft_consensus.cc:2781] T c9aa43ee18124167bf85018fb2bf1e49 P 8f77615a1c5e4961862b33a057d44c3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:51:28.983628 2021 raft_consensus.cc:482] T c9aa43ee18124167bf85018fb2bf1e49 P 8f77615a1c5e4961862b33a057d44c3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:51:28.983721 2021 raft_consensus.cc:3037] T c9aa43ee18124167bf85018fb2bf1e49 P 8f77615a1c5e4961862b33a057d44c3c [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:28.985819 2021 raft_consensus.cc:504] T c9aa43ee18124167bf85018fb2bf1e49 P 8f77615a1c5e4961862b33a057d44c3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36dde0a7a9114639bc27dca417b288cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46659 } } peers { permanent_uuid: "8f77615a1c5e4961862b33a057d44c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34683 } } peers { permanent_uuid: "f9dce3d411234778a20049dc1d077dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42325 } }
I20260929 15:51:28.986423 2021 leader_election.cc:288] T c9aa43ee18124167bf85018fb2bf1e49 P 8f77615a1c5e4961862b33a057d44c3c [CANDIDATE]: Term 1 election: Requested vote from peers 36dde0a7a9114639bc27dca417b288cc (127.0.97.1:46659), f9dce3d411234778a20049dc1d077dd2 (127.0.97.3:42325)
I20260929 15:51:28.986862 1672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9aa43ee18124167bf85018fb2bf1e49" candidate_uuid: "8f77615a1c5e4961862b33a057d44c3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36dde0a7a9114639bc27dca417b288cc"
I20260929 15:51:28.986883 1950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9aa43ee18124167bf85018fb2bf1e49" candidate_uuid: "8f77615a1c5e4961862b33a057d44c3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9dce3d411234778a20049dc1d077dd2"
I20260929 15:51:28.987066 1950 raft_consensus.cc:3037] T c9aa43ee18124167bf85018fb2bf1e49 P f9dce3d411234778a20049dc1d077dd2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:28.987110 1672 raft_consensus.cc:3037] T c9aa43ee18124167bf85018fb2bf1e49 P 36dde0a7a9114639bc27dca417b288cc [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:28.990374 1672 raft_consensus.cc:2454] T c9aa43ee18124167bf85018fb2bf1e49 P 36dde0a7a9114639bc27dca417b288cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f77615a1c5e4961862b33a057d44c3c in term 1.
I20260929 15:51:28.990389 1950 raft_consensus.cc:2454] T c9aa43ee18124167bf85018fb2bf1e49 P f9dce3d411234778a20049dc1d077dd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f77615a1c5e4961862b33a057d44c3c in term 1.
I20260929 15:51:28.990732 1738 leader_election.cc:302] T c9aa43ee18124167bf85018fb2bf1e49 P 8f77615a1c5e4961862b33a057d44c3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f77615a1c5e4961862b33a057d44c3c, f9dce3d411234778a20049dc1d077dd2; no voters:
I20260929 15:51:28.991014 2021 raft_consensus.cc:2781] T c9aa43ee18124167bf85018fb2bf1e49 P 8f77615a1c5e4961862b33a057d44c3c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:51:28.991348 2021 raft_consensus.cc:686] T c9aa43ee18124167bf85018fb2bf1e49 P 8f77615a1c5e4961862b33a057d44c3c [term 1 LEADER]: Becoming Leader. State: Replica: 8f77615a1c5e4961862b33a057d44c3c, State: Running, Role: LEADER
I20260929 15:51:28.991685 2021 consensus_queue.cc:237] T c9aa43ee18124167bf85018fb2bf1e49 P 8f77615a1c5e4961862b33a057d44c3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36dde0a7a9114639bc27dca417b288cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46659 } } peers { permanent_uuid: "8f77615a1c5e4961862b33a057d44c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34683 } } peers { permanent_uuid: "f9dce3d411234778a20049dc1d077dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42325 } }
I20260929 15:51:28.993567 2063 tablet_replica.cc:439] T c9aa43ee18124167bf85018fb2bf1e49 P 8f77615a1c5e4961862b33a057d44c3c: TxnStatusTablet state changed. Reason: New leader 8f77615a1c5e4961862b33a057d44c3c. Latest consensus state: current_term: 1 leader_uuid: "8f77615a1c5e4961862b33a057d44c3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36dde0a7a9114639bc27dca417b288cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46659 } } peers { permanent_uuid: "8f77615a1c5e4961862b33a057d44c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34683 } } peers { permanent_uuid: "f9dce3d411234778a20049dc1d077dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42325 } } }
I20260929 15:51:28.993930 2063 tablet_replica.cc:442] T c9aa43ee18124167bf85018fb2bf1e49 P 8f77615a1c5e4961862b33a057d44c3c: This TxnStatusTablet replica's current role is: LEADER
I20260929 15:51:28.994614 2065 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 15:51:28.994843 2065 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 15:51:28.995393 1510 catalog_manager.cc:5495] T c9aa43ee18124167bf85018fb2bf1e49 P 8f77615a1c5e4961862b33a057d44c3c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f77615a1c5e4961862b33a057d44c3c (127.0.97.2). New cstate: current_term: 1 leader_uuid: "8f77615a1c5e4961862b33a057d44c3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36dde0a7a9114639bc27dca417b288cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f77615a1c5e4961862b33a057d44c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9dce3d411234778a20049dc1d077dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42325 } health_report { overall_health: UNKNOWN } } }
I20260929 15:51:29.036525 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:29.030029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34683 (local address 127.0.0.1:34160)
0929 15:51:29.030266 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:29.030287 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:29.030415 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:29.030905 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:29.030908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:29.030918 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:51:29.031061 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:29.031068 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:29.032495 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:29.032499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:29.033667 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:29.033679 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:29.034664 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:29.034669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:29.034915 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:29.034975 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:51:29.036060 (+ 1085us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:51:29.036078 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 15:51:29.036155 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 15:51:29.036282 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 15:51:29.036566 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:29.030069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34160 (local address 127.0.97.2:34683)
0929 15:51:29.030258 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:29.030265 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:29.030371 (+ 106us) server_negotiation.cc:407] Connection header received
0929 15:51:29.030613 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:29.030620 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:29.030718 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:29.030871 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:51:29.031199 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:29.032349 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:29.033815 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:29.034500 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:29.034642 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:29.035165 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:51:29.035933 (+ 768us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:51:29.036068 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:29.036200 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 15:51:29.036283 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 15:51:29.041567 1950 raft_consensus.cc:1261] T c9aa43ee18124167bf85018fb2bf1e49 P f9dce3d411234778a20049dc1d077dd2 [term 1 FOLLOWER]: Refusing update from remote peer 8f77615a1c5e4961862b33a057d44c3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:51:29.041836 1672 raft_consensus.cc:1261] T c9aa43ee18124167bf85018fb2bf1e49 P 36dde0a7a9114639bc27dca417b288cc [term 1 FOLLOWER]: Refusing update from remote peer 8f77615a1c5e4961862b33a057d44c3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:51:29.042295 2063 consensus_queue.cc:1035] T c9aa43ee18124167bf85018fb2bf1e49 P 8f77615a1c5e4961862b33a057d44c3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9dce3d411234778a20049dc1d077dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:51:29.042703 2021 consensus_queue.cc:1035] T c9aa43ee18124167bf85018fb2bf1e49 P 8f77615a1c5e4961862b33a057d44c3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "36dde0a7a9114639bc27dca417b288cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:51:29.049422 2020 tablet_replica.cc:439] T c9aa43ee18124167bf85018fb2bf1e49 P f9dce3d411234778a20049dc1d077dd2: TxnStatusTablet state changed. Reason: New leader 8f77615a1c5e4961862b33a057d44c3c. Latest consensus state: current_term: 1 leader_uuid: "8f77615a1c5e4961862b33a057d44c3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36dde0a7a9114639bc27dca417b288cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46659 } } peers { permanent_uuid: "8f77615a1c5e4961862b33a057d44c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34683 } } peers { permanent_uuid: "f9dce3d411234778a20049dc1d077dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42325 } } }
I20260929 15:51:29.049777 2020 tablet_replica.cc:442] T c9aa43ee18124167bf85018fb2bf1e49 P f9dce3d411234778a20049dc1d077dd2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:51:29.051370 2068 mvcc.cc:205] Tried to move back new op lower bound from 7334695276707000320 to 7334695276513202176. Current Snapshot: MvccSnapshot[applied={T|T < 7334695276707000320}]
I20260929 15:51:29.053978 2020 tablet_replica.cc:439] T c9aa43ee18124167bf85018fb2bf1e49 P f9dce3d411234778a20049dc1d077dd2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8f77615a1c5e4961862b33a057d44c3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36dde0a7a9114639bc27dca417b288cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46659 } } peers { permanent_uuid: "8f77615a1c5e4961862b33a057d44c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34683 } } peers { permanent_uuid: "f9dce3d411234778a20049dc1d077dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42325 } } }
I20260929 15:51:29.054253 2020 tablet_replica.cc:442] T c9aa43ee18124167bf85018fb2bf1e49 P f9dce3d411234778a20049dc1d077dd2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:51:29.055713 2063 tablet_replica.cc:439] T c9aa43ee18124167bf85018fb2bf1e49 P 8f77615a1c5e4961862b33a057d44c3c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8f77615a1c5e4961862b33a057d44c3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36dde0a7a9114639bc27dca417b288cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46659 } } peers { permanent_uuid: "8f77615a1c5e4961862b33a057d44c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34683 } } peers { permanent_uuid: "f9dce3d411234778a20049dc1d077dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42325 } } }
I20260929 15:51:29.056006 2063 tablet_replica.cc:442] T c9aa43ee18124167bf85018fb2bf1e49 P 8f77615a1c5e4961862b33a057d44c3c: This TxnStatusTablet replica's current role is: LEADER
I20260929 15:51:29.061790 2025 tablet_replica.cc:439] T c9aa43ee18124167bf85018fb2bf1e49 P 36dde0a7a9114639bc27dca417b288cc: TxnStatusTablet state changed. Reason: New leader 8f77615a1c5e4961862b33a057d44c3c. Latest consensus state: current_term: 1 leader_uuid: "8f77615a1c5e4961862b33a057d44c3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36dde0a7a9114639bc27dca417b288cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46659 } } peers { permanent_uuid: "8f77615a1c5e4961862b33a057d44c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34683 } } peers { permanent_uuid: "f9dce3d411234778a20049dc1d077dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42325 } } }
I20260929 15:51:29.062079 2025 tablet_replica.cc:442] T c9aa43ee18124167bf85018fb2bf1e49 P 36dde0a7a9114639bc27dca417b288cc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:51:29.063978 2074 tablet_replica.cc:439] T c9aa43ee18124167bf85018fb2bf1e49 P 8f77615a1c5e4961862b33a057d44c3c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8f77615a1c5e4961862b33a057d44c3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36dde0a7a9114639bc27dca417b288cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46659 } } peers { permanent_uuid: "8f77615a1c5e4961862b33a057d44c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34683 } } peers { permanent_uuid: "f9dce3d411234778a20049dc1d077dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42325 } } }
I20260929 15:51:29.064349 2074 tablet_replica.cc:442] T c9aa43ee18124167bf85018fb2bf1e49 P 8f77615a1c5e4961862b33a057d44c3c: This TxnStatusTablet replica's current role is: LEADER
I20260929 15:51:29.066929 2025 tablet_replica.cc:439] T c9aa43ee18124167bf85018fb2bf1e49 P 36dde0a7a9114639bc27dca417b288cc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8f77615a1c5e4961862b33a057d44c3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36dde0a7a9114639bc27dca417b288cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46659 } } peers { permanent_uuid: "8f77615a1c5e4961862b33a057d44c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34683 } } peers { permanent_uuid: "f9dce3d411234778a20049dc1d077dd2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42325 } } }
I20260929 15:51:29.067190 2025 tablet_replica.cc:442] T c9aa43ee18124167bf85018fb2bf1e49 P 36dde0a7a9114639bc27dca417b288cc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:51:29.078047 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:29.070448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40028 (local address 127.0.97.3:42325)
0929 15:51:29.070670 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:29.070681 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:29.071216 (+ 535us) server_negotiation.cc:407] Connection header received
0929 15:51:29.072009 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:29.072015 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:29.072164 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:29.072311 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:51:29.072792 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:29.073772 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:29.075755 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:29.076171 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:29.076319 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:29.076766 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:51:29.077436 (+ 670us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:51:29.077589 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:29.077673 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 15:51:29.077754 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 15:51:29.094520 2081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:29.088649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34683 (local address 127.0.97.3:40987)
0929 15:51:29.089293 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:29.089309 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:29.089485 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:29.089802 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:29.089805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:29.089814 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:51:29.090082 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:29.090089 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:29.091437 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:29.091441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:29.092476 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:29.092487 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:29.093534 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:29.093539 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:29.093646 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:29.093696 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:51:29.094089 (+ 393us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:51:29.094096 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 15:51:29.094153 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 15:51:29.094242 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":370,"threads_started":1}
I20260929 15:51:29.094561 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:29.088665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40987 (local address 127.0.97.2:34683)
0929 15:51:29.088854 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:29.088860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:29.089440 (+ 580us) server_negotiation.cc:407] Connection header received
0929 15:51:29.089603 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:29.089609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:29.089696 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:29.089801 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:51:29.090208 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:29.091304 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:29.092612 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:29.093376 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:29.093505 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:29.093854 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:51:29.093996 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:51:29.094084 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:29.094198 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 15:51:29.094276 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 15:51:29.102895 2020 consensus_queue.cc:1035] T 6610c21fbef04604bd0013f5a0cced2a P f9dce3d411234778a20049dc1d077dd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36dde0a7a9114639bc27dca417b288cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:51:29.103322 1811 raft_consensus.cc:1261] T 6610c21fbef04604bd0013f5a0cced2a P 8f77615a1c5e4961862b33a057d44c3c [term 1 FOLLOWER]: Refusing update from remote peer f9dce3d411234778a20049dc1d077dd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:51:29.103845 2020 consensus_queue.cc:1035] T 6610c21fbef04604bd0013f5a0cced2a P f9dce3d411234778a20049dc1d077dd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f77615a1c5e4961862b33a057d44c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:51:29.143776 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:29.136671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50443 (local address 127.0.97.3:42325)
0929 15:51:29.136826 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:29.136831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:29.137514 (+ 683us) server_negotiation.cc:407] Connection header received
0929 15:51:29.137722 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:29.137727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:29.137821 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:29.137933 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:51:29.138861 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:29.139990 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:29.141494 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:29.142159 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:29.142283 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:29.143092 (+ 809us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:51:29.143205 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:51:29.143300 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:29.143403 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 15:51:29.143474 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 15:51:29.144140 2087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:29.136734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42325 (local address 127.0.97.2:50443)
0929 15:51:29.137403 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:29.137421 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:29.137591 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:29.137946 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:29.137951 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:29.137962 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:51:29.138677 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:29.138689 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:29.140122 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:29.140126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:29.141315 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:29.141327 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:29.142645 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:29.142653 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:29.142841 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:29.142905 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:51:29.143300 (+ 395us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:51:29.143312 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 15:51:29.143387 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 15:51:29.143509 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":349,"threads_started":1}
I20260929 15:51:29.179785 1510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:51:29.180037 1510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:51:29.182725 1510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6f7e0dcaa3c4ecd9658ee15fb3594ad: Sending DeleteTablet for 3 replicas of tablet 6610c21fbef04604bd0013f5a0cced2a
I20260929 15:51:29.183413 1646 tablet_service.cc:1514] Processing DeleteTablet for tablet 6610c21fbef04604bd0013f5a0cced2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:51:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42208
I20260929 15:51:29.183768 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 6610c21fbef04604bd0013f5a0cced2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:51:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40016
I20260929 15:51:29.184513 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet 6610c21fbef04604bd0013f5a0cced2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:51:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34156
I20260929 15:51:29.200641 2091 tablet_replica.cc:330] T 6610c21fbef04604bd0013f5a0cced2a P 36dde0a7a9114639bc27dca417b288cc: stopping tablet replica
I20260929 15:51:29.201047 2091 raft_consensus.cc:2229] T 6610c21fbef04604bd0013f5a0cced2a P 36dde0a7a9114639bc27dca417b288cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:51:29.201417 2091 raft_consensus.cc:2258] T 6610c21fbef04604bd0013f5a0cced2a P 36dde0a7a9114639bc27dca417b288cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:51:29.203101 2091 ts_tablet_manager.cc:1905] T 6610c21fbef04604bd0013f5a0cced2a P 36dde0a7a9114639bc27dca417b288cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:51:29.209834 2093 tablet_replica.cc:330] T 6610c21fbef04604bd0013f5a0cced2a P 8f77615a1c5e4961862b33a057d44c3c: stopping tablet replica
I20260929 15:51:29.210134 2093 raft_consensus.cc:2229] T 6610c21fbef04604bd0013f5a0cced2a P 8f77615a1c5e4961862b33a057d44c3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:51:29.210412 2093 raft_consensus.cc:2258] T 6610c21fbef04604bd0013f5a0cced2a P 8f77615a1c5e4961862b33a057d44c3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:51:29.211053 2092 tablet_replica.cc:330] T 6610c21fbef04604bd0013f5a0cced2a P f9dce3d411234778a20049dc1d077dd2: stopping tablet replica
I20260929 15:51:29.211278 2092 raft_consensus.cc:2229] T 6610c21fbef04604bd0013f5a0cced2a P f9dce3d411234778a20049dc1d077dd2 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:51:29.211865 2092 raft_consensus.cc:2258] T 6610c21fbef04604bd0013f5a0cced2a P f9dce3d411234778a20049dc1d077dd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:51:29.211913 2091 ts_tablet_manager.cc:1918] T 6610c21fbef04604bd0013f5a0cced2a P 36dde0a7a9114639bc27dca417b288cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:51:29.212147 2091 log.cc:1196] T 6610c21fbef04604bd0013f5a0cced2a P 36dde0a7a9114639bc27dca417b288cc: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-0/wal/wals/6610c21fbef04604bd0013f5a0cced2a
I20260929 15:51:29.212651 2091 ts_tablet_manager.cc:1939] T 6610c21fbef04604bd0013f5a0cced2a P 36dde0a7a9114639bc27dca417b288cc: Deleting consensus metadata
I20260929 15:51:29.213050 2093 ts_tablet_manager.cc:1905] T 6610c21fbef04604bd0013f5a0cced2a P 8f77615a1c5e4961862b33a057d44c3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:51:29.213896 1498 catalog_manager.cc:4841] TS 36dde0a7a9114639bc27dca417b288cc (127.0.97.1:46659): tablet 6610c21fbef04604bd0013f5a0cced2a (table test-table [id=069ce536d1124596bfc5a9f7e922cb7e]) successfully deleted
I20260929 15:51:29.218366 2092 ts_tablet_manager.cc:1905] T 6610c21fbef04604bd0013f5a0cced2a P f9dce3d411234778a20049dc1d077dd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:51:29.218722 2093 ts_tablet_manager.cc:1918] T 6610c21fbef04604bd0013f5a0cced2a P 8f77615a1c5e4961862b33a057d44c3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:51:29.218851 2093 log.cc:1196] T 6610c21fbef04604bd0013f5a0cced2a P 8f77615a1c5e4961862b33a057d44c3c: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-1/wal/wals/6610c21fbef04604bd0013f5a0cced2a
I20260929 15:51:29.219708 2093 ts_tablet_manager.cc:1939] T 6610c21fbef04604bd0013f5a0cced2a P 8f77615a1c5e4961862b33a057d44c3c: Deleting consensus metadata
I20260929 15:51:29.220855 1497 catalog_manager.cc:4841] TS 8f77615a1c5e4961862b33a057d44c3c (127.0.97.2:34683): tablet 6610c21fbef04604bd0013f5a0cced2a (table test-table [id=069ce536d1124596bfc5a9f7e922cb7e]) successfully deleted
I20260929 15:51:29.224612 2092 ts_tablet_manager.cc:1918] T 6610c21fbef04604bd0013f5a0cced2a P f9dce3d411234778a20049dc1d077dd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:51:29.224756 2092 log.cc:1196] T 6610c21fbef04604bd0013f5a0cced2a P f9dce3d411234778a20049dc1d077dd2: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697075657932-388-0/minicluster-data/ts-2/wal/wals/6610c21fbef04604bd0013f5a0cced2a
I20260929 15:51:29.225220 2092 ts_tablet_manager.cc:1939] T 6610c21fbef04604bd0013f5a0cced2a P f9dce3d411234778a20049dc1d077dd2: Deleting consensus metadata
Sep 29 15:51:29 dist-test-slave-r0ck krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697087, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 15:51:29.226311 1496 catalog_manager.cc:4841] TS f9dce3d411234778a20049dc1d077dd2 (127.0.97.3:42325): tablet 6610c21fbef04604bd0013f5a0cced2a (table test-table [id=069ce536d1124596bfc5a9f7e922cb7e]) successfully deleted
I20260929 15:51:29.232741 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:29.205734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42226 (local address 127.0.97.1:46659)
0929 15:51:29.205908 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:29.205914 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:29.210367 (+ 4453us) server_negotiation.cc:407] Connection header received
0929 15:51:29.210887 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:29.210893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:29.210978 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:29.211049 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:29.213129 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:29.214222 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:29.216085 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:29.216729 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:29.216823 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:29.227199 (+ 10376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:29.227212 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:29.227226 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:29.227263 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:29.230085 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:29.230777 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:29.230784 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:29.230786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:29.230909 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:29.231266 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:29.231272 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:29.231276 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:29.231609 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:29.231971 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:29.232197 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 15:51:29.232409 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260929 15:51:29.233578 1712 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42226
I20260929 15:51:29.254946 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:29.236569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43356 (local address 127.0.97.62:33219)
0929 15:51:29.236795 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:29.236802 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:29.243005 (+ 6203us) server_negotiation.cc:407] Connection header received
0929 15:51:29.243200 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:29.243206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:29.243304 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:29.243402 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:29.244654 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:29.245414 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:29.246609 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:29.247043 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:29.247186 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:29.249414 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:29.249421 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:29.249428 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:29.249464 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:29.252410 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:29.253028 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:29.253034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:29.253038 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:29.253155 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:29.253586 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:29.253592 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:29.253595 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:29.253860 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:29.254099 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:29.254399 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 15:51:29.254608 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
W20260929 15:51:29.255517 1510 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43356
I20260929 15:51:29.259419 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 1582
I20260929 15:51:29.273702 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:29.260122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42238 (local address 127.0.97.1:46659)
0929 15:51:29.260300 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:29.260308 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:29.261803 (+ 1495us) server_negotiation.cc:407] Connection header received
0929 15:51:29.261985 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:29.261991 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:29.262065 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:29.262163 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:29.263202 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:29.263884 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:29.265038 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:29.265511 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:29.265635 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:29.267770 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:29.267776 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:29.267780 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:29.267819 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:29.270703 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:29.271503 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:29.271510 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:29.271514 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:29.271641 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:29.272075 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:29.272082 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:29.272085 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:29.272405 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:29.272737 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:29.272979 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 15:51:29.273183 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260929 15:51:29.274219 1712 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42238
W20260929 15:51:29.274737 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 1582: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:51:29.274923 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 1582
I20260929 15:51:29.297361 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 1721
Sep 29 15:51:29 dist-test-slave-r0ck krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697087, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 15:51:29.313795 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:29.298126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34162 (local address 127.0.97.2:34683)
0929 15:51:29.298301 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:29.298307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:29.298451 (+ 144us) server_negotiation.cc:407] Connection header received
0929 15:51:29.298657 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:29.298665 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:29.298739 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:29.298843 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:29.300183 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:29.300919 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:29.302314 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:29.302767 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:29.302897 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:29.307105 (+ 4208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:29.307115 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:29.307127 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:29.307168 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:29.310506 (+ 3338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:29.311331 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:29.311339 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:29.311342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:29.311518 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:29.311957 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:29.311962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:29.311966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:29.312432 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:29.312735 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:29.313336 (+ 601us) server_negotiation.cc:299] Negotiation successful
0929 15:51:29.313548 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260929 15:51:29.314972 1851 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34162
W20260929 15:51:29.315630 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 1721: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:51:29.315791 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 1721
I20260929 15:51:29.341575 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 1860
Sep 29 15:51:29 dist-test-slave-r0ck krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697087, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 15:51:29.359519 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:29.342319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40040 (local address 127.0.97.3:42325)
0929 15:51:29.342498 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:29.342507 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:29.342643 (+ 136us) server_negotiation.cc:407] Connection header received
0929 15:51:29.342839 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:29.342847 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:29.342931 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:29.343040 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:29.344584 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:29.345621 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:29.347421 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:29.347839 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:29.347930 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:29.352835 (+ 4905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:29.352852 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:29.352872 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:29.352928 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:29.356834 (+ 3906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:29.357527 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:29.357536 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:29.357540 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:29.357699 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:29.358018 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:29.358028 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:29.358033 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:29.358488 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:29.358774 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:29.358956 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 15:51:29.359176 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260929 15:51:29.360230 1990 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40040
W20260929 15:51:29.360859 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 1860: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:51:29.360997 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 1860
I20260929 15:51:29.379117 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 1479
I20260929 15:51:29.380131 1551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:51:29.492198 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 1479
2026-09-29T15:51:29Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4831 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:51:29 dist-test-slave-r0ck krb5kdc[2102](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:51:29 dist-test-slave-r0ck krb5kdc[2102](info): set up 2 sockets
Sep 29 15:51:29 dist-test-slave-r0ck krb5kdc[2102](info): commencing operation
krb5kdc: starting...
W20260929 15:51:30.573504 388 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:51:30 dist-test-slave-r0ck krb5kdc[2102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:51:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:51:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:51:30.760850 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36055
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:51:31.076088 2115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:51:31.076445 2115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:51:31.076630 2115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:51:31.083859 2115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:51:31.083964 2115 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 15:51:31.084033 2115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:51:31.084124 2115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:51:31.084208 2115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:51:31.084314 2115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:51:31.094241 2115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36055
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:51:31.095480 2115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:51:31.096865 2115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:51:31.108664 2120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:31.108649 2121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:31.108707 2123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:51:31.109571 2115 server_base.cc:1034] running on GCE node
I20260929 15:51:31.110316 2115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:51:31.111662 2115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:51:31.112910 2115 hybrid_clock.cc:648] HybridClock initialized: now 1790697091112832 us; error 51 us; skew 500 ppm
I20260929 15:51:31.113229 2115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:51:31 dist-test-slave-r0ck krb5kdc[2102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697091, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:51:31.116325 2115 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:51:31.117805 2115 webserver.cc:460] Webserver started at http://127.0.97.62:35609/ using document root <none> and password file <none>
I20260929 15:51:31.118476 2115 fs_manager.cc:359] Metadata directory not provided
I20260929 15:51:31.118603 2115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:51:31.118906 2115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:51:31.121790 2115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "8087fc375014469989c805e973593c1b"
format_stamp: "Formatted at 2026-09-29 15:51:31 on dist-test-slave-r0ck"
server_key: "7ea768d0fe081e8b34e3af099f1c2e99"
server_key_iv: "0014264c6e9526c172e0e0182e317bd0"
server_key_version: "encryptionkey@0"
I20260929 15:51:31.122561 2115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "8087fc375014469989c805e973593c1b"
format_stamp: "Formatted at 2026-09-29 15:51:31 on dist-test-slave-r0ck"
server_key: "7ea768d0fe081e8b34e3af099f1c2e99"
server_key_iv: "0014264c6e9526c172e0e0182e317bd0"
server_key_version: "encryptionkey@0"
I20260929 15:51:31.127430 2115 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 15:51:31.130722 2130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:31.131985 2115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:51:31.132138 2115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "8087fc375014469989c805e973593c1b"
format_stamp: "Formatted at 2026-09-29 15:51:31 on dist-test-slave-r0ck"
server_key: "7ea768d0fe081e8b34e3af099f1c2e99"
server_key_iv: "0014264c6e9526c172e0e0182e317bd0"
server_key_version: "encryptionkey@0"
I20260929 15:51:31.132313 2115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:51:31.143536 2115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:51:31.147805 2115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:51:31.148075 2115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:51:31.168907 2115 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36055
I20260929 15:51:31.168928 2191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36055 every 8 connection(s)
I20260929 15:51:31.170243 2115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:51:31.173933 2192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:31.178284 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 2115
I20260929 15:51:31.178498 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:51:31.179085 388 external_mini_cluster.cc:1428] Setting key 548d42fad42234a11ec98523b53604b3
I20260929 15:51:31.184015 2192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8087fc375014469989c805e973593c1b: Bootstrap starting.
I20260929 15:51:31.187515 2192 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8087fc375014469989c805e973593c1b: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:31.188673 2192 log.cc:824] T 00000000000000000000000000000000 P 8087fc375014469989c805e973593c1b: Log is configured to *not* fsync() on all Append() calls
I20260929 15:51:31.191931 2192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8087fc375014469989c805e973593c1b: No bootstrap required, opened a new log
Sep 29 15:51:31 dist-test-slave-r0ck krb5kdc[2102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:51:31.200090 2192 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8087fc375014469989c805e973593c1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8087fc375014469989c805e973593c1b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36055 } }
I20260929 15:51:31.200487 2192 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8087fc375014469989c805e973593c1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:31.200578 2192 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8087fc375014469989c805e973593c1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8087fc375014469989c805e973593c1b, State: Initialized, Role: FOLLOWER
I20260929 15:51:31.201126 2192 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8087fc375014469989c805e973593c1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8087fc375014469989c805e973593c1b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36055 } }
I20260929 15:51:31.201413 2192 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8087fc375014469989c805e973593c1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:51:31.201572 2192 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8087fc375014469989c805e973593c1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:51:31.201725 2195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:31.180970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42830 (local address 127.0.97.62:36055)
0929 15:51:31.181805 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:31.181816 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:31.181857 (+ 41us) server_negotiation.cc:407] Connection header received
0929 15:51:31.183477 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:31.183498 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:31.183837 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:31.184180 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:31.185911 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:31.187137 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:31.188922 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:31.190020 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:31.190140 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:31.194651 (+ 4511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:31.194662 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:31.194675 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:31.194705 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:31.197655 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:31.198311 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:31.198317 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:31.198322 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:31.198500 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:31.198864 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:31.198869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:31.198872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:31.199334 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:31.199627 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:31.200059 (+ 432us) server_negotiation.cc:299] Negotiation successful
0929 15:51:31.200481 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":378,"threads_started":1}
I20260929 15:51:31.201761 2192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8087fc375014469989c805e973593c1b [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:31.204099 2192 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8087fc375014469989c805e973593c1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8087fc375014469989c805e973593c1b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36055 } }
I20260929 15:51:31.204618 2192 leader_election.cc:302] T 00000000000000000000000000000000 P 8087fc375014469989c805e973593c1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8087fc375014469989c805e973593c1b; no voters:
I20260929 15:51:31.205077 2192 leader_election.cc:288] T 00000000000000000000000000000000 P 8087fc375014469989c805e973593c1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:51:31.205220 2197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8087fc375014469989c805e973593c1b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:51:31.205904 2197 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8087fc375014469989c805e973593c1b [term 1 LEADER]: Becoming Leader. State: Replica: 8087fc375014469989c805e973593c1b, State: Running, Role: LEADER
I20260929 15:51:31.206357 2197 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8087fc375014469989c805e973593c1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8087fc375014469989c805e973593c1b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36055 } }
I20260929 15:51:31.207050 2192 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8087fc375014469989c805e973593c1b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:51:31.208845 2198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8087fc375014469989c805e973593c1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8087fc375014469989c805e973593c1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8087fc375014469989c805e973593c1b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36055 } } }
I20260929 15:51:31.208953 2199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8087fc375014469989c805e973593c1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8087fc375014469989c805e973593c1b. Latest consensus state: current_term: 1 leader_uuid: "8087fc375014469989c805e973593c1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8087fc375014469989c805e973593c1b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36055 } } }
I20260929 15:51:31.209261 2198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8087fc375014469989c805e973593c1b [sys.catalog]: This master's current role is: LEADER
I20260929 15:51:31.209933 2199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8087fc375014469989c805e973593c1b [sys.catalog]: This master's current role is: LEADER
I20260929 15:51:31.210887 2202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:51:31.215250 2202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:51:31.223528 2202 catalog_manager.cc:1347] Generated new cluster ID: 13fcf16405fc4979850b756fd540049c
I20260929 15:51:31.223649 2202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:51:31.248415 2202 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:51:31.249817 2202 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:51:31.262496 2202 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8087fc375014469989c805e973593c1b: Generated new TSK 0
I20260929 15:51:31.263386 2202 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 15:51:31.307089 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36055
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:51:31.622300 2218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:51:31.622598 2218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:51:31.622771 2218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:51:31.629797 2218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:51:31.629940 2218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:51:31.630090 2218 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 15:51:31.630175 2218 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 15:51:31.630306 2218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:51:31.639897 2218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:36055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:51:31.641099 2218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:51:31.642408 2218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:51:31.654827 2226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:31.654806 2224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:31.655370 2223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:51:31.655625 2218 server_base.cc:1034] running on GCE node
I20260929 15:51:31.656214 2218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:51:31.657028 2218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:51:31.658218 2218 hybrid_clock.cc:648] HybridClock initialized: now 1790697091658187 us; error 26 us; skew 500 ppm
I20260929 15:51:31.658526 2218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:51:31 dist-test-slave-r0ck krb5kdc[2102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697091, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:51:31.661634 2218 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:51:31.662931 2218 webserver.cc:460] Webserver started at http://127.0.97.1:43869/ using document root <none> and password file <none>
I20260929 15:51:31.663569 2218 fs_manager.cc:359] Metadata directory not provided
I20260929 15:51:31.663695 2218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:51:31.664002 2218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:51:31.666770 2218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-0/data/instance:
uuid: "9d5c2b08b9854b64a4e1491ca9ae1021"
format_stamp: "Formatted at 2026-09-29 15:51:31 on dist-test-slave-r0ck"
server_key: "533e5d0618385d683a91cff514a99329"
server_key_iv: "64338fc38e798ef5472861d67aeadff8"
server_key_version: "encryptionkey@0"
I20260929 15:51:31.667621 2218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "9d5c2b08b9854b64a4e1491ca9ae1021"
format_stamp: "Formatted at 2026-09-29 15:51:31 on dist-test-slave-r0ck"
server_key: "533e5d0618385d683a91cff514a99329"
server_key_iv: "64338fc38e798ef5472861d67aeadff8"
server_key_version: "encryptionkey@0"
I20260929 15:51:31.672325 2218 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 15:51:31.675498 2233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:31.676726 2218 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 15:51:31.676893 2218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-0/wal
uuid: "9d5c2b08b9854b64a4e1491ca9ae1021"
format_stamp: "Formatted at 2026-09-29 15:51:31 on dist-test-slave-r0ck"
server_key: "533e5d0618385d683a91cff514a99329"
server_key_iv: "64338fc38e798ef5472861d67aeadff8"
server_key_version: "encryptionkey@0"
I20260929 15:51:31.677095 2218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:51:31.701220 2218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:51:31.705555 2218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:51:31.705830 2218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:51:31.708566 2218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:51:31.708678 2218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:31.708791 2218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:51:31.708926 2218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 15:51:31 dist-test-slave-r0ck krb5kdc[2102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697091, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:51:31.736553 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:31.709058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51517 (local address 127.0.97.62:36055)
0929 15:51:31.713851 (+ 4793us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:31.713864 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:31.714688 (+ 824us) server_negotiation.cc:407] Connection header received
0929 15:51:31.717540 (+ 2852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:31.717548 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:31.717655 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:31.717793 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:31.719372 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:31.720810 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:31.723148 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:31.723905 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:31.724043 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:31.728743 (+ 4700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:31.728748 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:31.728752 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:31.728783 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:31.732596 (+ 3813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:31.733454 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:31.733464 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:31.733470 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:31.733648 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:31.734157 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:31.734165 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:31.734171 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:31.734637 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:31.734823 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:31.735664 (+ 841us) server_negotiation.cc:299] Negotiation successful
0929 15:51:31.735985 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4683,"thread_start_us":4598,"threads_started":1}
I20260929 15:51:31.737277 2243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:31.709776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36055 (local address 127.0.97.1:51517)
0929 15:51:31.714527 (+ 4751us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:31.714577 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:31.717161 (+ 2584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:31.718049 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:31.718057 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:31.718593 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:51:31.719211 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:31.719228 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:31.720982 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:31.720985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:31.722989 (+ 2004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:31.723007 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:31.724185 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:31.724197 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:31.724369 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:31.725034 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:51:31.725060 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:51:31.728477 (+ 3417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:51:31.732757 (+ 4280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:31.732764 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:31.732778 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:31.733190 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:31.733766 (+ 576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:31.733773 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:31.733782 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:31.734022 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:31.734875 (+ 853us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:51:31.734882 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:51:31.735225 (+ 343us) client_negotiation.cc:769] Sending connection context
0929 15:51:31.735512 (+ 287us) client_negotiation.cc:240] Negotiation successful
0929 15:51:31.736003 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4500,"thread_start_us":4191,"threads_started":1}
I20260929 15:51:31.757999 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:31.743301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43367 (local address 127.0.97.62:36055)
0929 15:51:31.743527 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:31.743537 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:31.744040 (+ 503us) server_negotiation.cc:407] Connection header received
0929 15:51:31.744174 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:31.744182 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:31.744279 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:31.744397 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:31.745228 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:31.746482 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:31.748387 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:31.749334 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:31.749533 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:31.752165 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:31.752170 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:31.752173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:31.752203 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:31.755107 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:31.756037 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:31.756045 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:31.756049 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:31.756206 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:31.756559 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:31.756565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:31.756569 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:31.756854 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:31.757002 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:31.757426 (+ 424us) server_negotiation.cc:299] Negotiation successful
0929 15:51:31.757644 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 15:51:31.758100 2302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:31.743320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36055 (local address 127.0.97.1:43367)
0929 15:51:31.743933 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:31.743959 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:31.744066 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:31.744424 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:31.744427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:31.744843 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:51:31.745099 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:31.745107 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:31.746678 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:31.746689 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:31.748201 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:31.748221 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:31.749765 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:31.749782 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:31.750020 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:31.750768 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:51:31.750787 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:51:31.751962 (+ 1175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:51:31.755235 (+ 3273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:31.755241 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:31.755247 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:31.755828 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:31.756313 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:31.756320 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:31.756322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:31.756446 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:31.756986 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:51:31.756991 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:51:31.757159 (+ 168us) client_negotiation.cc:769] Sending connection context
0929 15:51:31.757309 (+ 150us) client_negotiation.cc:240] Negotiation successful
0929 15:51:31.757663 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":339,"threads_started":1}
I20260929 15:51:31.768769 2218 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37035
I20260929 15:51:31.768836 2353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37035 every 8 connection(s)
I20260929 15:51:31.770048 2218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:51:31.771974 2354 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36055
I20260929 15:51:31.772204 2354 heartbeater.cc:464] Registering TS with master...
I20260929 15:51:31.772819 2354 heartbeater.cc:511] Master 127.0.97.62:36055 requested a full tablet report, sending...
I20260929 15:51:31.774551 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 2218
I20260929 15:51:31.774737 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:51:31.774777 2147 ts_manager.cc:194] Registered new tserver with Master: 9d5c2b08b9854b64a4e1491ca9ae1021 (127.0.97.1:37035)
I20260929 15:51:31.775358 388 external_mini_cluster.cc:1428] Setting key 7914772c3212774210bbe5df3e83b903
I20260929 15:51:31.776455 2147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:51517
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 15:51:31.812875 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36055
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:51:32.119259 2358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:51:32.119524 2358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:51:32.119678 2358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:51:32.126714 2358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:51:32.126806 2358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:51:32.126976 2358 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 15:51:32.127033 2358 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 15:51:32.127156 2358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:51:32.136406 2358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:36055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:51:32.137650 2358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:51:32.138916 2358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:51:32.151208 2366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:32.151243 2363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:32.151185 2364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:51:32.152522 2358 server_base.cc:1034] running on GCE node
I20260929 15:51:32.153105 2358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:51:32.154040 2358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:51:32.155225 2358 hybrid_clock.cc:648] HybridClock initialized: now 1790697092155169 us; error 61 us; skew 500 ppm
I20260929 15:51:32.155515 2358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:51:32 dist-test-slave-r0ck krb5kdc[2102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:51:32.158536 2358 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:51:32.159785 2358 webserver.cc:460] Webserver started at http://127.0.97.2:38551/ using document root <none> and password file <none>
I20260929 15:51:32.160418 2358 fs_manager.cc:359] Metadata directory not provided
I20260929 15:51:32.160511 2358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:51:32.160748 2358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:51:32.163501 2358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-1/data/instance:
uuid: "25bedb4d9c1f46aeaa1256eb6708adaa"
format_stamp: "Formatted at 2026-09-29 15:51:32 on dist-test-slave-r0ck"
server_key: "c691e17220cf764cfa1d23bb2e150c01"
server_key_iv: "7fae0845b8320591b637d802750eb96f"
server_key_version: "encryptionkey@0"
I20260929 15:51:32.164189 2358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "25bedb4d9c1f46aeaa1256eb6708adaa"
format_stamp: "Formatted at 2026-09-29 15:51:32 on dist-test-slave-r0ck"
server_key: "c691e17220cf764cfa1d23bb2e150c01"
server_key_iv: "7fae0845b8320591b637d802750eb96f"
server_key_version: "encryptionkey@0"
I20260929 15:51:32.169282 2358 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260929 15:51:32.172643 2373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:32.174048 2358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:51:32.174216 2358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-1/wal
uuid: "25bedb4d9c1f46aeaa1256eb6708adaa"
format_stamp: "Formatted at 2026-09-29 15:51:32 on dist-test-slave-r0ck"
server_key: "c691e17220cf764cfa1d23bb2e150c01"
server_key_iv: "7fae0845b8320591b637d802750eb96f"
server_key_version: "encryptionkey@0"
I20260929 15:51:32.174407 2358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:51:32.199262 2358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:51:32.203150 2358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:51:32.203423 2358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:51:32.205981 2358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:51:32.206113 2358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:32.206240 2358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:51:32.206337 2358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 15:51:32 dist-test-slave-r0ck krb5kdc[2102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:51:32.229238 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:32.206567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39067 (local address 127.0.97.62:36055)
0929 15:51:32.206784 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:32.206791 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:32.208231 (+ 1440us) server_negotiation.cc:407] Connection header received
0929 15:51:32.210266 (+ 2035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:32.210275 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:32.210382 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:32.210500 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:32.212561 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:32.213653 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:32.216245 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:32.216874 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:32.217014 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:32.222775 (+ 5761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:32.222784 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:32.222790 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:32.222837 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:32.226185 (+ 3348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:32.226927 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:32.226933 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:32.226937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:32.227073 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:32.227513 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:32.227519 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:32.227522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:32.227786 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:32.227947 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:32.228650 (+ 703us) server_negotiation.cc:299] Negotiation successful
0929 15:51:32.228891 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 15:51:32.230271 2382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:32.207157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36055 (local address 127.0.97.2:39067)
0929 15:51:32.208045 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:32.208097 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:32.210015 (+ 1918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:32.210780 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:32.210790 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:32.211514 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:51:32.212387 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:32.212411 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:32.213812 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:32.213820 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:32.216055 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:32.216071 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:32.217106 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:32.217114 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:32.217472 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:32.218334 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:51:32.218385 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:51:32.222481 (+ 4096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:51:32.226333 (+ 3852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:32.226344 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:32.226360 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:32.226784 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:32.227184 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:32.227190 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:32.227194 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:32.227399 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:32.227927 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:51:32.227935 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:51:32.228279 (+ 344us) client_negotiation.cc:769] Sending connection context
0929 15:51:32.228553 (+ 274us) client_negotiation.cc:240] Negotiation successful
0929 15:51:32.229109 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":382,"threads_started":1}
I20260929 15:51:32.251065 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:32.237573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45591 (local address 127.0.97.62:36055)
0929 15:51:32.237745 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:32.237753 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:32.238673 (+ 920us) server_negotiation.cc:407] Connection header received
0929 15:51:32.238953 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:32.238958 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:32.239037 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:32.239137 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:32.240007 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:32.240703 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:32.242356 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:32.242767 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:32.242913 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:32.244661 (+ 1748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:32.244667 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:32.244671 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:32.244709 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:32.247765 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:32.248663 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:32.248671 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:32.248677 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:32.248845 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:32.249283 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:32.249290 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:32.249296 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:32.249725 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:32.249890 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:32.250416 (+ 526us) server_negotiation.cc:299] Negotiation successful
0929 15:51:32.250671 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 15:51:32.251467 2457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:32.237663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36055 (local address 127.0.97.2:45591)
0929 15:51:32.238559 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:32.238585 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:32.238753 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:32.239167 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:32.239171 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:32.239599 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:51:32.239870 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:32.239878 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:32.240845 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:32.240851 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:32.242213 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:32.242225 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:32.242951 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:32.242958 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:32.243129 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:32.243755 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:51:32.243772 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:51:32.244473 (+ 701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:51:32.247908 (+ 3435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:32.247913 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:32.247917 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:32.248538 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:32.249018 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:32.249028 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:32.249033 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:32.249164 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:32.249880 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:51:32.249886 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:51:32.250093 (+ 207us) client_negotiation.cc:769] Sending connection context
0929 15:51:32.250297 (+ 204us) client_negotiation.cc:240] Negotiation successful
0929 15:51:32.250817 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"thread_start_us":400,"threads_started":1}
I20260929 15:51:32.256893 2358 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33143
I20260929 15:51:32.256990 2492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33143 every 8 connection(s)
I20260929 15:51:32.258291 2358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:51:32.260285 2493 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36055
I20260929 15:51:32.260460 2493 heartbeater.cc:464] Registering TS with master...
I20260929 15:51:32.260998 2493 heartbeater.cc:511] Master 127.0.97.62:36055 requested a full tablet report, sending...
I20260929 15:51:32.262283 2147 ts_manager.cc:194] Registered new tserver with Master: 25bedb4d9c1f46aeaa1256eb6708adaa (127.0.97.2:33143)
I20260929 15:51:32.263293 2147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39067
I20260929 15:51:32.268613 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 2358
I20260929 15:51:32.268774 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:51:32.269217 388 external_mini_cluster.cc:1428] Setting key ecbbcb580ae55c66d0370991043f262b
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 15:51:32.308595 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36055
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:51:32.622471 2497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:51:32.622725 2497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:51:32.622823 2497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:51:32.629724 2497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:51:32.629817 2497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:51:32.629899 2497 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 15:51:32.629956 2497 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 15:51:32.630016 2497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:51:32.639430 2497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:36055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:51:32.640661 2497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:51:32.641974 2497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:51:32.655040 2505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:32.655067 2503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:32.655115 2502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:51:32.656190 2497 server_base.cc:1034] running on GCE node
I20260929 15:51:32.656729 2497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:51:32.657749 2497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:51:32.658977 2497 hybrid_clock.cc:648] HybridClock initialized: now 1790697092658894 us; error 69 us; skew 500 ppm
I20260929 15:51:32.659269 2497 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:51:32 dist-test-slave-r0ck krb5kdc[2102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:51:32.665478 2497 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:51:32.666787 2497 webserver.cc:460] Webserver started at http://127.0.97.3:36469/ using document root <none> and password file <none>
I20260929 15:51:32.667420 2497 fs_manager.cc:359] Metadata directory not provided
I20260929 15:51:32.667563 2497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:51:32.667853 2497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:51:32.670581 2497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-2/data/instance:
uuid: "32d80eed68624cbeb7750d84632e714c"
format_stamp: "Formatted at 2026-09-29 15:51:32 on dist-test-slave-r0ck"
server_key: "909b66edecdef72ee037550ce01f8812"
server_key_iv: "3b55c34be857ad087f6905c0e7a24923"
server_key_version: "encryptionkey@0"
I20260929 15:51:32.671315 2497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "32d80eed68624cbeb7750d84632e714c"
format_stamp: "Formatted at 2026-09-29 15:51:32 on dist-test-slave-r0ck"
server_key: "909b66edecdef72ee037550ce01f8812"
server_key_iv: "3b55c34be857ad087f6905c0e7a24923"
server_key_version: "encryptionkey@0"
I20260929 15:51:32.675958 2497 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 15:51:32.679309 2512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:32.680728 2497 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 15:51:32.680908 2497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-2/wal
uuid: "32d80eed68624cbeb7750d84632e714c"
format_stamp: "Formatted at 2026-09-29 15:51:32 on dist-test-slave-r0ck"
server_key: "909b66edecdef72ee037550ce01f8812"
server_key_iv: "3b55c34be857ad087f6905c0e7a24923"
server_key_version: "encryptionkey@0"
I20260929 15:51:32.681120 2497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:51:32.695040 2497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:51:32.698786 2497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:51:32.699080 2497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:51:32.701612 2497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:51:32.701733 2497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:32.701877 2497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:51:32.701978 2497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 15:51:32 dist-test-slave-r0ck krb5kdc[2102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:51:32.725987 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:32.701984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53849 (local address 127.0.97.62:36055)
0929 15:51:32.702196 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:32.702204 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:32.703728 (+ 1524us) server_negotiation.cc:407] Connection header received
0929 15:51:32.706710 (+ 2982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:32.706716 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:32.706809 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:32.706937 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:32.709319 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:32.710452 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:32.712085 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:32.712488 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:32.712621 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:32.718542 (+ 5921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:32.718553 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:32.718560 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:32.718614 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:32.722439 (+ 3825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:32.723233 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:32.723240 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:32.723243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:32.723413 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:32.724044 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:32.724047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:32.724049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:32.724291 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:32.724400 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:32.725555 (+ 1155us) server_negotiation.cc:299] Negotiation successful
0929 15:51:32.725724 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 15:51:32.728248 2521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:32.702657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36055 (local address 127.0.97.3:53849)
0929 15:51:32.703543 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:32.703588 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:32.706370 (+ 2782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:32.707389 (+ 1019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:32.707396 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:32.708200 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:51:32.709118 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:32.709151 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:32.710610 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:32.710616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:32.711929 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:32.711941 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:32.712729 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:32.712736 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:32.712960 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:32.714095 (+ 1135us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:51:32.714139 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:51:32.718091 (+ 3952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:51:32.722592 (+ 4501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:32.722602 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:32.722622 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:32.723110 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:32.723566 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:32.723575 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:32.723581 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:32.723910 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:32.724425 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:51:32.724437 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:51:32.725030 (+ 593us) client_negotiation.cc:769] Sending connection context
0929 15:51:32.725458 (+ 428us) client_negotiation.cc:240] Negotiation successful
0929 15:51:32.726213 (+ 755us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":352,"threads_started":1}
I20260929 15:51:32.750015 2582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:32.735093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36055 (local address 127.0.97.3:43553)
0929 15:51:32.735722 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:32.735741 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:32.735883 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:32.736232 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:32.736237 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:32.736665 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:51:32.736918 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:32.736926 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:32.738088 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:32.738094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:32.739803 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:32.739828 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:32.741099 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:32.741108 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:32.741259 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:32.741977 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:51:32.742006 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:51:32.743212 (+ 1206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:51:32.747496 (+ 4284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:32.747507 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:32.747513 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:32.747883 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:32.748313 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:32.748316 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:32.748318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:32.748422 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:32.749107 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:51:32.749110 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:51:32.749255 (+ 145us) client_negotiation.cc:769] Sending connection context
0929 15:51:32.749441 (+ 186us) client_negotiation.cc:240] Negotiation successful
0929 15:51:32.749691 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":363,"threads_started":1}
I20260929 15:51:32.750348 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:32.735004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43553 (local address 127.0.97.62:36055)
0929 15:51:32.735173 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:32.735177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:32.735818 (+ 641us) server_negotiation.cc:407] Connection header received
0929 15:51:32.736020 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:32.736026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:32.736092 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:32.736192 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:32.737058 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:32.737975 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:32.739975 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:32.740829 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:32.740982 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:32.743428 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:32.743438 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:32.743446 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:32.743504 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:32.747337 (+ 3833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:32.747997 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:32.748005 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:32.748012 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:32.748192 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:32.748525 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:32.748533 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:32.748539 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:32.748982 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:32.749152 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:32.749589 (+ 437us) server_negotiation.cc:299] Negotiation successful
0929 15:51:32.749874 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 15:51:32.759070 2497 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43617
I20260929 15:51:32.759116 2631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43617 every 8 connection(s)
I20260929 15:51:32.760288 2497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:51:32.762200 2632 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36055
I20260929 15:51:32.762424 2632 heartbeater.cc:464] Registering TS with master...
I20260929 15:51:32.763055 2632 heartbeater.cc:511] Master 127.0.97.62:36055 requested a full tablet report, sending...
I20260929 15:51:32.764284 2147 ts_manager.cc:194] Registered new tserver with Master: 32d80eed68624cbeb7750d84632e714c (127.0.97.3:43617)
I20260929 15:51:32.764678 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 2497
I20260929 15:51:32.764844 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:51:32.765204 2147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53849
I20260929 15:51:32.765264 388 external_mini_cluster.cc:1428] Setting key bab14cc7c6f4dd04ca1d7f26ca35a238
I20260929 15:51:32.768049 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:51:32.778925 2354 heartbeater.cc:503] Master 127.0.97.62:36055 was elected leader, sending a full tablet report...
Sep 29 15:51:32 dist-test-slave-r0ck krb5kdc[2102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 15:51:32 dist-test-slave-r0ck krb5kdc[2102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:51:32.811671 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:32.798454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42840 (local address 127.0.97.62:36055)
0929 15:51:32.798669 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:32.798676 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:32.798930 (+ 254us) server_negotiation.cc:407] Connection header received
0929 15:51:32.799132 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:32.799137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:32.799217 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:32.799322 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:32.800599 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:32.801519 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:32.802776 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:32.803206 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:32.803339 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:32.807092 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:32.807096 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:32.807100 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:32.807129 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:32.809707 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:32.810250 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:32.810254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:32.810256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:32.810358 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:32.810650 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:32.810653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:32.810655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:32.810872 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:32.811040 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:32.811261 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 15:51:32.811413 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 15:51:32.815478 2147 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:51:32.822453 2147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:51:32.847980 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:32.833945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37035 (local address 127.0.0.1:50954)
0929 15:51:32.835854 (+ 1909us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:32.835909 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:32.836161 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:32.840878 (+ 4717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:32.840885 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:32.840925 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:51:32.841424 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:32.841437 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:32.843088 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:32.843096 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:32.845175 (+ 2079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:32.845189 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:32.846914 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:32.846920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:32.847110 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:32.847175 (+ 65us) client_negotiation.cc:769] Sending connection context
0929 15:51:32.847344 (+ 169us) client_negotiation.cc:240] Negotiation successful
0929 15:51:32.847580 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1579,"thread_start_us":2507,"threads_started":1}
I20260929 15:51:32.848747 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:32.834458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50954 (local address 127.0.97.1:37035)
0929 15:51:32.840201 (+ 5743us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:32.840214 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:32.840247 (+ 33us) server_negotiation.cc:407] Connection header received
0929 15:51:32.840355 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:32.840366 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:32.840639 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:32.840848 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:51:32.841669 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:32.842956 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:32.845379 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:32.846717 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:32.846869 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:32.847094 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:32.848274 (+ 1180us) server_negotiation.cc:299] Negotiation successful
0929 15:51:32.848458 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5541,"thread_start_us":5402,"threads_started":1}
I20260929 15:51:32.849907 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:32.835117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59922 (local address 127.0.97.3:43617)
0929 15:51:32.838950 (+ 3833us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:32.838963 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:32.839007 (+ 44us) server_negotiation.cc:407] Connection header received
0929 15:51:32.839128 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:32.839139 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:32.839471 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:32.841918 (+ 2447us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:51:32.842211 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:32.843744 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:32.846372 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:32.847902 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:32.848032 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:32.849389 (+ 1357us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:32.849456 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 15:51:32.849601 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3602,"thread_start_us":3340,"threads_started":1}
I20260929 15:51:32.849916 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:32.834408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33143 (local address 127.0.0.1:54660)
0929 15:51:32.840881 (+ 6473us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:32.840898 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:32.841038 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:32.841961 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:32.841965 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:32.841990 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:51:32.842317 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:32.842328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:32.844032 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:32.844038 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:32.845869 (+ 1831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:32.845880 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:32.849087 (+ 3207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:32.849093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:32.849245 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:32.849276 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 15:51:32.849395 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 15:51:32.849522 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6326,"mutex_wait_us":20,"spinlock_wait_cycles":6400,"thread_start_us":6217,"threads_started":1}
I20260929 15:51:32.849959 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:32.834633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54660 (local address 127.0.97.2:33143)
0929 15:51:32.840485 (+ 5852us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:32.840492 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:32.841178 (+ 686us) server_negotiation.cc:407] Connection header received
0929 15:51:32.841293 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:32.841301 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:32.841558 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:32.841740 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:51:32.842506 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:32.843891 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:32.846067 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:32.848617 (+ 2550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:32.848746 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:32.848964 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:32.849485 (+ 521us) server_negotiation.cc:299] Negotiation successful
0929 15:51:32.849652 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5706,"thread_start_us":5577,"threads_started":1}
I20260929 15:51:32.852303 2284 tablet_service.cc:1467] Processing CreateTablet for tablet bdd799e6c9ed4123ae947f1086e70ab0 (DEFAULT_TABLE table=test-table [id=4eeb78f3898d4974822d8f4d689ba4a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:51:32.853669 2284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdd799e6c9ed4123ae947f1086e70ab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:32.856060 2423 tablet_service.cc:1467] Processing CreateTablet for tablet bdd799e6c9ed4123ae947f1086e70ab0 (DEFAULT_TABLE table=test-table [id=4eeb78f3898d4974822d8f4d689ba4a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:51:32.857275 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:32.834056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43617 (local address 127.0.0.1:59922)
0929 15:51:32.836736 (+ 2680us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:32.836758 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:32.836898 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:32.841156 (+ 4258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:32.841162 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:32.841192 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:51:32.841715 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:32.841726 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:32.843906 (+ 2180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:32.843912 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:32.846196 (+ 2284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:32.846209 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:32.848389 (+ 2180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:32.848400 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:32.848567 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:32.848600 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 15:51:32.848684 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 15:51:32.856880 (+ 8196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2536,"mutex_wait_us":52,"thread_start_us":1323,"threads_started":1}
I20260929 15:51:32.857455 2423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdd799e6c9ed4123ae947f1086e70ab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:32.860292 2562 tablet_service.cc:1467] Processing CreateTablet for tablet bdd799e6c9ed4123ae947f1086e70ab0 (DEFAULT_TABLE table=test-table [id=4eeb78f3898d4974822d8f4d689ba4a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:51:32.861438 2562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdd799e6c9ed4123ae947f1086e70ab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:32.869283 2651 tablet_bootstrap.cc:492] T bdd799e6c9ed4123ae947f1086e70ab0 P 9d5c2b08b9854b64a4e1491ca9ae1021: Bootstrap starting.
I20260929 15:51:32.871366 2652 tablet_bootstrap.cc:492] T bdd799e6c9ed4123ae947f1086e70ab0 P 25bedb4d9c1f46aeaa1256eb6708adaa: Bootstrap starting.
I20260929 15:51:32.873216 2651 tablet_bootstrap.cc:654] T bdd799e6c9ed4123ae947f1086e70ab0 P 9d5c2b08b9854b64a4e1491ca9ae1021: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:32.873236 2653 tablet_bootstrap.cc:492] T bdd799e6c9ed4123ae947f1086e70ab0 P 32d80eed68624cbeb7750d84632e714c: Bootstrap starting.
I20260929 15:51:32.874583 2651 log.cc:824] T bdd799e6c9ed4123ae947f1086e70ab0 P 9d5c2b08b9854b64a4e1491ca9ae1021: Log is configured to *not* fsync() on all Append() calls
I20260929 15:51:32.874677 2652 tablet_bootstrap.cc:654] T bdd799e6c9ed4123ae947f1086e70ab0 P 25bedb4d9c1f46aeaa1256eb6708adaa: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:32.875881 2652 log.cc:824] T bdd799e6c9ed4123ae947f1086e70ab0 P 25bedb4d9c1f46aeaa1256eb6708adaa: Log is configured to *not* fsync() on all Append() calls
I20260929 15:51:32.876612 2653 tablet_bootstrap.cc:654] T bdd799e6c9ed4123ae947f1086e70ab0 P 32d80eed68624cbeb7750d84632e714c: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:32.877323 2651 tablet_bootstrap.cc:492] T bdd799e6c9ed4123ae947f1086e70ab0 P 9d5c2b08b9854b64a4e1491ca9ae1021: No bootstrap required, opened a new log
I20260929 15:51:32.877646 2651 ts_tablet_manager.cc:1397] T bdd799e6c9ed4123ae947f1086e70ab0 P 9d5c2b08b9854b64a4e1491ca9ae1021: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260929 15:51:32.877718 2653 log.cc:824] T bdd799e6c9ed4123ae947f1086e70ab0 P 32d80eed68624cbeb7750d84632e714c: Log is configured to *not* fsync() on all Append() calls
I20260929 15:51:32.879096 2652 tablet_bootstrap.cc:492] T bdd799e6c9ed4123ae947f1086e70ab0 P 25bedb4d9c1f46aeaa1256eb6708adaa: No bootstrap required, opened a new log
I20260929 15:51:32.879396 2652 ts_tablet_manager.cc:1397] T bdd799e6c9ed4123ae947f1086e70ab0 P 25bedb4d9c1f46aeaa1256eb6708adaa: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260929 15:51:32.885823 2651 raft_consensus.cc:348] T bdd799e6c9ed4123ae947f1086e70ab0 P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } }
I20260929 15:51:32.886179 2651 raft_consensus.cc:374] T bdd799e6c9ed4123ae947f1086e70ab0 P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:32.886276 2651 raft_consensus.cc:729] T bdd799e6c9ed4123ae947f1086e70ab0 P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d5c2b08b9854b64a4e1491ca9ae1021, State: Initialized, Role: FOLLOWER
I20260929 15:51:32.886874 2651 consensus_queue.cc:260] T bdd799e6c9ed4123ae947f1086e70ab0 P 9d5c2b08b9854b64a4e1491ca9ae1021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } }
I20260929 15:51:32.889233 2651 ts_tablet_manager.cc:1428] T bdd799e6c9ed4123ae947f1086e70ab0 P 9d5c2b08b9854b64a4e1491ca9ae1021: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
I20260929 15:51:32.889823 2653 tablet_bootstrap.cc:492] T bdd799e6c9ed4123ae947f1086e70ab0 P 32d80eed68624cbeb7750d84632e714c: No bootstrap required, opened a new log
I20260929 15:51:32.890095 2653 ts_tablet_manager.cc:1397] T bdd799e6c9ed4123ae947f1086e70ab0 P 32d80eed68624cbeb7750d84632e714c: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.008s
I20260929 15:51:32.891755 2652 raft_consensus.cc:348] T bdd799e6c9ed4123ae947f1086e70ab0 P 25bedb4d9c1f46aeaa1256eb6708adaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } }
I20260929 15:51:32.892110 2652 raft_consensus.cc:374] T bdd799e6c9ed4123ae947f1086e70ab0 P 25bedb4d9c1f46aeaa1256eb6708adaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:32.892177 2652 raft_consensus.cc:729] T bdd799e6c9ed4123ae947f1086e70ab0 P 25bedb4d9c1f46aeaa1256eb6708adaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25bedb4d9c1f46aeaa1256eb6708adaa, State: Initialized, Role: FOLLOWER
I20260929 15:51:32.892706 2652 consensus_queue.cc:260] T bdd799e6c9ed4123ae947f1086e70ab0 P 25bedb4d9c1f46aeaa1256eb6708adaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } }
I20260929 15:51:32.893774 2493 heartbeater.cc:503] Master 127.0.97.62:36055 was elected leader, sending a full tablet report...
I20260929 15:51:32.894428 2652 ts_tablet_manager.cc:1428] T bdd799e6c9ed4123ae947f1086e70ab0 P 25bedb4d9c1f46aeaa1256eb6708adaa: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260929 15:51:32.898996 2653 raft_consensus.cc:348] T bdd799e6c9ed4123ae947f1086e70ab0 P 32d80eed68624cbeb7750d84632e714c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } }
I20260929 15:51:32.899335 2653 raft_consensus.cc:374] T bdd799e6c9ed4123ae947f1086e70ab0 P 32d80eed68624cbeb7750d84632e714c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:32.899449 2653 raft_consensus.cc:729] T bdd799e6c9ed4123ae947f1086e70ab0 P 32d80eed68624cbeb7750d84632e714c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32d80eed68624cbeb7750d84632e714c, State: Initialized, Role: FOLLOWER
I20260929 15:51:32.899971 2653 consensus_queue.cc:260] T bdd799e6c9ed4123ae947f1086e70ab0 P 32d80eed68624cbeb7750d84632e714c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } }
I20260929 15:51:32.900991 2632 heartbeater.cc:503] Master 127.0.97.62:36055 was elected leader, sending a full tablet report...
I20260929 15:51:32.901311 2653 ts_tablet_manager.cc:1428] T bdd799e6c9ed4123ae947f1086e70ab0 P 32d80eed68624cbeb7750d84632e714c: Time spent starting tablet: real 0.011s user 0.010s sys 0.001s
I20260929 15:51:32.928399 2659 raft_consensus.cc:482] T bdd799e6c9ed4123ae947f1086e70ab0 P 32d80eed68624cbeb7750d84632e714c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:51:32.928740 2659 raft_consensus.cc:504] T bdd799e6c9ed4123ae947f1086e70ab0 P 32d80eed68624cbeb7750d84632e714c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } }
I20260929 15:51:32.930466 2659 leader_election.cc:288] T bdd799e6c9ed4123ae947f1086e70ab0 P 32d80eed68624cbeb7750d84632e714c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d5c2b08b9854b64a4e1491ca9ae1021 (127.0.97.1:37035), 25bedb4d9c1f46aeaa1256eb6708adaa (127.0.97.2:33143)
I20260929 15:51:32.936683 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:32.930863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44723 (local address 127.0.97.1:37035)
0929 15:51:32.931087 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:32.931093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:32.931330 (+ 237us) server_negotiation.cc:407] Connection header received
0929 15:51:32.931412 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:32.931421 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:32.931523 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:32.931617 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:51:32.932216 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:32.933269 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:32.934882 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:32.935738 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:32.935894 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:32.936019 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:32.936304 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 15:51:32.936424 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 15:51:32.936805 2521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:32.930720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37035 (local address 127.0.97.3:44723)
0929 15:51:32.931062 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:32.931083 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:32.931214 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:32.931782 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:32.931789 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:32.931835 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:51:32.932054 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:32.932068 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:32.933408 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:32.933417 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:32.934714 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:32.934725 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:32.935903 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:32.935908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:32.936146 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:32.936171 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 15:51:32.936279 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 15:51:32.936437 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":23}
I20260929 15:51:32.937330 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:32.930832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37399 (local address 127.0.97.2:33143)
0929 15:51:32.931276 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:32.931283 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:32.931953 (+ 670us) server_negotiation.cc:407] Connection header received
0929 15:51:32.932169 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:32.932173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:32.932265 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:32.932360 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:51:32.932992 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:32.933878 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:32.935594 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:32.936392 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:32.936554 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:32.936665 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:32.937040 (+ 375us) server_negotiation.cc:299] Negotiation successful
0929 15:51:32.937164 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286}
I20260929 15:51:32.937932 2660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:32.930767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33143 (local address 127.0.97.3:37399)
0929 15:51:32.931833 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:32.931850 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:32.931967 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:32.932392 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:32.932396 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:32.932420 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:51:32.932753 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:32.932763 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:32.934005 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:32.934012 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:32.935439 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:32.935450 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:32.936634 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:32.936641 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:32.936852 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:32.936880 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 15:51:32.937570 (+ 690us) client_negotiation.cc:240] Negotiation successful
0929 15:51:32.937687 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":930,"mutex_wait_us":41,"thread_start_us":744,"threads_started":1}
I20260929 15:51:32.937786 2309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdd799e6c9ed4123ae947f1086e70ab0" candidate_uuid: "32d80eed68624cbeb7750d84632e714c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" is_pre_election: true
I20260929 15:51:32.938288 2309 raft_consensus.cc:2454] T bdd799e6c9ed4123ae947f1086e70ab0 P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32d80eed68624cbeb7750d84632e714c in term 0.
I20260929 15:51:32.938898 2513 leader_election.cc:302] T bdd799e6c9ed4123ae947f1086e70ab0 P 32d80eed68624cbeb7750d84632e714c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32d80eed68624cbeb7750d84632e714c, 9d5c2b08b9854b64a4e1491ca9ae1021; no voters:
I20260929 15:51:32.938817 2443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdd799e6c9ed4123ae947f1086e70ab0" candidate_uuid: "32d80eed68624cbeb7750d84632e714c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" is_pre_election: true
I20260929 15:51:32.939297 2659 raft_consensus.cc:2781] T bdd799e6c9ed4123ae947f1086e70ab0 P 32d80eed68624cbeb7750d84632e714c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:51:32.939352 2443 raft_consensus.cc:2454] T bdd799e6c9ed4123ae947f1086e70ab0 P 25bedb4d9c1f46aeaa1256eb6708adaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32d80eed68624cbeb7750d84632e714c in term 0.
I20260929 15:51:32.939451 2659 raft_consensus.cc:482] T bdd799e6c9ed4123ae947f1086e70ab0 P 32d80eed68624cbeb7750d84632e714c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:51:32.939548 2659 raft_consensus.cc:3037] T bdd799e6c9ed4123ae947f1086e70ab0 P 32d80eed68624cbeb7750d84632e714c [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:32.942005 2659 raft_consensus.cc:504] T bdd799e6c9ed4123ae947f1086e70ab0 P 32d80eed68624cbeb7750d84632e714c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } }
I20260929 15:51:32.942533 2659 leader_election.cc:288] T bdd799e6c9ed4123ae947f1086e70ab0 P 32d80eed68624cbeb7750d84632e714c [CANDIDATE]: Term 1 election: Requested vote from peers 9d5c2b08b9854b64a4e1491ca9ae1021 (127.0.97.1:37035), 25bedb4d9c1f46aeaa1256eb6708adaa (127.0.97.2:33143)
I20260929 15:51:32.942955 2309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdd799e6c9ed4123ae947f1086e70ab0" candidate_uuid: "32d80eed68624cbeb7750d84632e714c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021"
I20260929 15:51:32.943017 2443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdd799e6c9ed4123ae947f1086e70ab0" candidate_uuid: "32d80eed68624cbeb7750d84632e714c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa"
I20260929 15:51:32.943177 2309 raft_consensus.cc:3037] T bdd799e6c9ed4123ae947f1086e70ab0 P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:32.943184 2443 raft_consensus.cc:3037] T bdd799e6c9ed4123ae947f1086e70ab0 P 25bedb4d9c1f46aeaa1256eb6708adaa [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:32.946324 2309 raft_consensus.cc:2454] T bdd799e6c9ed4123ae947f1086e70ab0 P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32d80eed68624cbeb7750d84632e714c in term 1.
I20260929 15:51:32.946324 2443 raft_consensus.cc:2454] T bdd799e6c9ed4123ae947f1086e70ab0 P 25bedb4d9c1f46aeaa1256eb6708adaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32d80eed68624cbeb7750d84632e714c in term 1.
I20260929 15:51:32.946732 2514 leader_election.cc:302] T bdd799e6c9ed4123ae947f1086e70ab0 P 32d80eed68624cbeb7750d84632e714c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25bedb4d9c1f46aeaa1256eb6708adaa, 32d80eed68624cbeb7750d84632e714c; no voters:
I20260929 15:51:32.946996 2659 raft_consensus.cc:2781] T bdd799e6c9ed4123ae947f1086e70ab0 P 32d80eed68624cbeb7750d84632e714c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:51:32.947396 2659 raft_consensus.cc:686] T bdd799e6c9ed4123ae947f1086e70ab0 P 32d80eed68624cbeb7750d84632e714c [term 1 LEADER]: Becoming Leader. State: Replica: 32d80eed68624cbeb7750d84632e714c, State: Running, Role: LEADER
I20260929 15:51:32.947820 2659 consensus_queue.cc:237] T bdd799e6c9ed4123ae947f1086e70ab0 P 32d80eed68624cbeb7750d84632e714c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } }
I20260929 15:51:32.952011 2147 catalog_manager.cc:5495] T bdd799e6c9ed4123ae947f1086e70ab0 P 32d80eed68624cbeb7750d84632e714c reported cstate change: term changed from 0 to 1, leader changed from <none> to 32d80eed68624cbeb7750d84632e714c (127.0.97.3). New cstate: current_term: 1 leader_uuid: "32d80eed68624cbeb7750d84632e714c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } health_report { overall_health: UNKNOWN } } }
Sep 29 15:51:32 dist-test-slave-r0ck krb5kdc[2102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697091, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:51:32.992980 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:32.980097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42848 (local address 127.0.97.62:36055)
0929 15:51:32.980260 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:32.980264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:32.980774 (+ 510us) server_negotiation.cc:407] Connection header received
0929 15:51:32.980951 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:32.980956 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:32.981033 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:32.981146 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:32.982352 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:32.983031 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:32.984300 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:32.984715 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:32.984876 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:32.988036 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:32.988041 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:32.988052 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:32.988081 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:32.990438 (+ 2357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:32.991098 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:32.991106 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:32.991112 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:32.991234 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:32.991582 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:32.991590 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:32.991593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:32.991820 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:32.991963 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:32.992360 (+ 397us) server_negotiation.cc:299] Negotiation successful
0929 15:51:32.992673 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":128,"server-negotiator.queue_time_us":58}
I20260929 15:51:32.993013 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:32.980098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36055 (local address 127.0.0.1:42848)
0929 15:51:32.980668 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:32.980687 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:32.980809 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:32.981185 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:32.981191 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:32.981941 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:51:32.982199 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:32.982208 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:32.983155 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:32.983163 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:32.984166 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:32.984178 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:32.984900 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:32.984906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:32.985080 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:32.985765 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:51:32.985787 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:51:32.987866 (+ 2079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:51:32.990567 (+ 2701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:32.990579 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:32.990587 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:32.990983 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:32.991335 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:32.991340 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:32.991343 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:32.991483 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:32.991947 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:51:32.991954 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:51:32.992112 (+ 158us) client_negotiation.cc:769] Sending connection context
0929 15:51:32.992269 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 15:51:32.992550 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":324,"threads_started":1}
I20260929 15:51:32.997975 2147 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42848:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 15:51:32.999181 2147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:51:33.006125 2284 tablet_service.cc:1467] Processing CreateTablet for tablet 9e741703042847e8845daf7cebedd16b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2f94fca58de947b4bbd6b1ee6298572f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 15:51:33.006422 2562 tablet_service.cc:1467] Processing CreateTablet for tablet 9e741703042847e8845daf7cebedd16b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2f94fca58de947b4bbd6b1ee6298572f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 15:51:33.006271 2423 tablet_service.cc:1467] Processing CreateTablet for tablet 9e741703042847e8845daf7cebedd16b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2f94fca58de947b4bbd6b1ee6298572f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 15:51:33.007181 2284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e741703042847e8845daf7cebedd16b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:33.007189 2423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e741703042847e8845daf7cebedd16b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:33.007195 2562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e741703042847e8845daf7cebedd16b. 1 dirs total, 0 dirs full, 0 dirs failed
W20260929 15:51:33.010202 2494 tablet.cc:2369] T bdd799e6c9ed4123ae947f1086e70ab0 P 25bedb4d9c1f46aeaa1256eb6708adaa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:51:33.011706 2633 tablet.cc:2369] T bdd799e6c9ed4123ae947f1086e70ab0 P 32d80eed68624cbeb7750d84632e714c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:51:33.013566 2651 tablet_bootstrap.cc:492] T 9e741703042847e8845daf7cebedd16b P 9d5c2b08b9854b64a4e1491ca9ae1021: Bootstrap starting.
I20260929 15:51:33.014611 2652 tablet_bootstrap.cc:492] T 9e741703042847e8845daf7cebedd16b P 25bedb4d9c1f46aeaa1256eb6708adaa: Bootstrap starting.
I20260929 15:51:33.015184 2651 tablet_bootstrap.cc:654] T 9e741703042847e8845daf7cebedd16b P 9d5c2b08b9854b64a4e1491ca9ae1021: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:33.015376 2653 tablet_bootstrap.cc:492] T 9e741703042847e8845daf7cebedd16b P 32d80eed68624cbeb7750d84632e714c: Bootstrap starting.
I20260929 15:51:33.016427 2652 tablet_bootstrap.cc:654] T 9e741703042847e8845daf7cebedd16b P 25bedb4d9c1f46aeaa1256eb6708adaa: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:33.017002 2653 tablet_bootstrap.cc:654] T 9e741703042847e8845daf7cebedd16b P 32d80eed68624cbeb7750d84632e714c: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:33.018332 2651 tablet_bootstrap.cc:492] T 9e741703042847e8845daf7cebedd16b P 9d5c2b08b9854b64a4e1491ca9ae1021: No bootstrap required, opened a new log
I20260929 15:51:33.018518 2651 ts_tablet_manager.cc:1397] T 9e741703042847e8845daf7cebedd16b P 9d5c2b08b9854b64a4e1491ca9ae1021: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 15:51:33.019170 2653 tablet_bootstrap.cc:492] T 9e741703042847e8845daf7cebedd16b P 32d80eed68624cbeb7750d84632e714c: No bootstrap required, opened a new log
I20260929 15:51:33.019320 2653 ts_tablet_manager.cc:1397] T 9e741703042847e8845daf7cebedd16b P 32d80eed68624cbeb7750d84632e714c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 15:51:33.019522 2651 raft_consensus.cc:348] T 9e741703042847e8845daf7cebedd16b P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } }
I20260929 15:51:33.019860 2651 raft_consensus.cc:374] T 9e741703042847e8845daf7cebedd16b P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:33.019963 2651 raft_consensus.cc:729] T 9e741703042847e8845daf7cebedd16b P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d5c2b08b9854b64a4e1491ca9ae1021, State: Initialized, Role: FOLLOWER
I20260929 15:51:33.020002 2653 raft_consensus.cc:348] T 9e741703042847e8845daf7cebedd16b P 32d80eed68624cbeb7750d84632e714c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } }
I20260929 15:51:33.020241 2653 raft_consensus.cc:374] T 9e741703042847e8845daf7cebedd16b P 32d80eed68624cbeb7750d84632e714c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:33.020329 2653 raft_consensus.cc:729] T 9e741703042847e8845daf7cebedd16b P 32d80eed68624cbeb7750d84632e714c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32d80eed68624cbeb7750d84632e714c, State: Initialized, Role: FOLLOWER
I20260929 15:51:33.020220 2651 consensus_queue.cc:260] T 9e741703042847e8845daf7cebedd16b P 9d5c2b08b9854b64a4e1491ca9ae1021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } }
I20260929 15:51:33.020764 2657 tablet_replica.cc:439] T 9e741703042847e8845daf7cebedd16b P 9d5c2b08b9854b64a4e1491ca9ae1021: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } }
I20260929 15:51:33.021041 2651 ts_tablet_manager.cc:1428] T 9e741703042847e8845daf7cebedd16b P 9d5c2b08b9854b64a4e1491ca9ae1021: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 15:51:33.021138 2657 tablet_replica.cc:442] T 9e741703042847e8845daf7cebedd16b P 9d5c2b08b9854b64a4e1491ca9ae1021: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:51:33.021808 2653 consensus_queue.cc:260] T 9e741703042847e8845daf7cebedd16b P 32d80eed68624cbeb7750d84632e714c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } }
W20260929 15:51:33.022280 2355 tablet.cc:2369] T 9e741703042847e8845daf7cebedd16b P 9d5c2b08b9854b64a4e1491ca9ae1021: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:51:33.022238 2659 tablet_replica.cc:439] T 9e741703042847e8845daf7cebedd16b P 32d80eed68624cbeb7750d84632e714c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } }
I20260929 15:51:33.022487 2659 tablet_replica.cc:442] T 9e741703042847e8845daf7cebedd16b P 32d80eed68624cbeb7750d84632e714c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:51:33.022538 2652 tablet_bootstrap.cc:492] T 9e741703042847e8845daf7cebedd16b P 25bedb4d9c1f46aeaa1256eb6708adaa: No bootstrap required, opened a new log
I20260929 15:51:33.022675 2652 ts_tablet_manager.cc:1397] T 9e741703042847e8845daf7cebedd16b P 25bedb4d9c1f46aeaa1256eb6708adaa: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260929 15:51:33.023567 2653 ts_tablet_manager.cc:1428] T 9e741703042847e8845daf7cebedd16b P 32d80eed68624cbeb7750d84632e714c: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260929 15:51:33.023615 2652 raft_consensus.cc:348] T 9e741703042847e8845daf7cebedd16b P 25bedb4d9c1f46aeaa1256eb6708adaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } }
I20260929 15:51:33.023874 2652 raft_consensus.cc:374] T 9e741703042847e8845daf7cebedd16b P 25bedb4d9c1f46aeaa1256eb6708adaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:33.023972 2652 raft_consensus.cc:729] T 9e741703042847e8845daf7cebedd16b P 25bedb4d9c1f46aeaa1256eb6708adaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25bedb4d9c1f46aeaa1256eb6708adaa, State: Initialized, Role: FOLLOWER
I20260929 15:51:33.024227 2652 consensus_queue.cc:260] T 9e741703042847e8845daf7cebedd16b P 25bedb4d9c1f46aeaa1256eb6708adaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } }
I20260929 15:51:33.024849 2652 ts_tablet_manager.cc:1428] T 9e741703042847e8845daf7cebedd16b P 25bedb4d9c1f46aeaa1256eb6708adaa: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 15:51:33.024674 2658 tablet_replica.cc:439] T 9e741703042847e8845daf7cebedd16b P 25bedb4d9c1f46aeaa1256eb6708adaa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } }
I20260929 15:51:33.025070 2658 tablet_replica.cc:442] T 9e741703042847e8845daf7cebedd16b P 25bedb4d9c1f46aeaa1256eb6708adaa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:51:33.042696 2657 raft_consensus.cc:482] T 9e741703042847e8845daf7cebedd16b P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:51:33.042896 2657 raft_consensus.cc:504] T 9e741703042847e8845daf7cebedd16b P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } }
I20260929 15:51:33.044363 2657 leader_election.cc:288] T 9e741703042847e8845daf7cebedd16b P 9d5c2b08b9854b64a4e1491ca9ae1021 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25bedb4d9c1f46aeaa1256eb6708adaa (127.0.97.2:33143), 32d80eed68624cbeb7750d84632e714c (127.0.97.3:43617)
I20260929 15:51:33.052124 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:33.044826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46559 (local address 127.0.97.3:43617)
0929 15:51:33.044989 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:33.044995 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:33.045747 (+ 752us) server_negotiation.cc:407] Connection header received
0929 15:51:33.045931 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:33.045936 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:33.046005 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:33.046101 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:51:33.046765 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:33.047615 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:33.049652 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:33.050856 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:33.050968 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:33.051063 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:33.051715 (+ 652us) server_negotiation.cc:299] Negotiation successful
0929 15:51:33.051869 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 15:51:33.052147 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:33.044819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44013 (local address 127.0.97.2:33143)
0929 15:51:33.044989 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:33.044996 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:33.045814 (+ 818us) server_negotiation.cc:407] Connection header received
0929 15:51:33.045925 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:33.045930 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:33.046005 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:33.046101 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:51:33.046935 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:33.047756 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:33.049603 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:33.050813 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:33.050962 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:33.051053 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:33.051715 (+ 662us) server_negotiation.cc:299] Negotiation successful
0929 15:51:33.051848 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 15:51:33.052290 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:33.044807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43617 (local address 127.0.97.1:46559)
0929 15:51:33.045613 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:33.045633 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:33.045795 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:33.046135 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:33.046138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:33.046162 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:51:33.046578 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:33.046590 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:33.047743 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:33.047748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:33.049464 (+ 1716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:33.049480 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:33.051118 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:33.051123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:33.051395 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:33.051535 (+ 140us) client_negotiation.cc:769] Sending connection context
0929 15:51:33.051634 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 15:51:33.051812 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"mutex_wait_us":150,"thread_start_us":440,"threads_started":1}
I20260929 15:51:33.052703 2673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:33.044862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33143 (local address 127.0.97.1:44013)
0929 15:51:33.045706 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:33.045720 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:33.045811 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:33.046302 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:33.046307 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:33.046390 (+ 83us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:51:33.046779 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:33.046793 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:33.047903 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:33.047908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:33.049454 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:33.049466 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:33.051113 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:33.051120 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:33.051391 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:33.051535 (+ 144us) client_negotiation.cc:769] Sending connection context
0929 15:51:33.051634 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 15:51:33.051812 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"mutex_wait_us":168,"spinlock_wait_cycles":21888,"thread_start_us":534,"threads_started":1}
I20260929 15:51:33.053244 2587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e741703042847e8845daf7cebedd16b" candidate_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32d80eed68624cbeb7750d84632e714c" is_pre_election: true
I20260929 15:51:33.053401 2443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e741703042847e8845daf7cebedd16b" candidate_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" is_pre_election: true
I20260929 15:51:33.053638 2443 raft_consensus.cc:2454] T 9e741703042847e8845daf7cebedd16b P 25bedb4d9c1f46aeaa1256eb6708adaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d5c2b08b9854b64a4e1491ca9ae1021 in term 0.
I20260929 15:51:33.053695 2587 raft_consensus.cc:2454] T 9e741703042847e8845daf7cebedd16b P 32d80eed68624cbeb7750d84632e714c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d5c2b08b9854b64a4e1491ca9ae1021 in term 0.
I20260929 15:51:33.054105 2235 leader_election.cc:302] T 9e741703042847e8845daf7cebedd16b P 9d5c2b08b9854b64a4e1491ca9ae1021 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25bedb4d9c1f46aeaa1256eb6708adaa, 9d5c2b08b9854b64a4e1491ca9ae1021; no voters:
I20260929 15:51:33.054510 2657 raft_consensus.cc:2781] T 9e741703042847e8845daf7cebedd16b P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:51:33.054657 2657 raft_consensus.cc:482] T 9e741703042847e8845daf7cebedd16b P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:51:33.054756 2657 raft_consensus.cc:3037] T 9e741703042847e8845daf7cebedd16b P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:33.056864 2657 raft_consensus.cc:504] T 9e741703042847e8845daf7cebedd16b P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } }
I20260929 15:51:33.057559 2657 leader_election.cc:288] T 9e741703042847e8845daf7cebedd16b P 9d5c2b08b9854b64a4e1491ca9ae1021 [CANDIDATE]: Term 1 election: Requested vote from peers 25bedb4d9c1f46aeaa1256eb6708adaa (127.0.97.2:33143), 32d80eed68624cbeb7750d84632e714c (127.0.97.3:43617)
I20260929 15:51:33.057943 2443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e741703042847e8845daf7cebedd16b" candidate_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa"
I20260929 15:51:33.058032 2587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e741703042847e8845daf7cebedd16b" candidate_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32d80eed68624cbeb7750d84632e714c"
I20260929 15:51:33.058161 2443 raft_consensus.cc:3037] T 9e741703042847e8845daf7cebedd16b P 25bedb4d9c1f46aeaa1256eb6708adaa [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:33.058226 2587 raft_consensus.cc:3037] T 9e741703042847e8845daf7cebedd16b P 32d80eed68624cbeb7750d84632e714c [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:33.060245 2443 raft_consensus.cc:2454] T 9e741703042847e8845daf7cebedd16b P 25bedb4d9c1f46aeaa1256eb6708adaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d5c2b08b9854b64a4e1491ca9ae1021 in term 1.
I20260929 15:51:33.060322 2587 raft_consensus.cc:2454] T 9e741703042847e8845daf7cebedd16b P 32d80eed68624cbeb7750d84632e714c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d5c2b08b9854b64a4e1491ca9ae1021 in term 1.
I20260929 15:51:33.060662 2235 leader_election.cc:302] T 9e741703042847e8845daf7cebedd16b P 9d5c2b08b9854b64a4e1491ca9ae1021 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25bedb4d9c1f46aeaa1256eb6708adaa, 9d5c2b08b9854b64a4e1491ca9ae1021; no voters:
I20260929 15:51:33.060977 2657 raft_consensus.cc:2781] T 9e741703042847e8845daf7cebedd16b P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:51:33.061453 2657 raft_consensus.cc:686] T 9e741703042847e8845daf7cebedd16b P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 1 LEADER]: Becoming Leader. State: Replica: 9d5c2b08b9854b64a4e1491ca9ae1021, State: Running, Role: LEADER
I20260929 15:51:33.061918 2657 consensus_queue.cc:237] T 9e741703042847e8845daf7cebedd16b P 9d5c2b08b9854b64a4e1491ca9ae1021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } }
I20260929 15:51:33.064901 2674 tablet_replica.cc:439] T 9e741703042847e8845daf7cebedd16b P 9d5c2b08b9854b64a4e1491ca9ae1021: TxnStatusTablet state changed. Reason: New leader 9d5c2b08b9854b64a4e1491ca9ae1021. Latest consensus state: current_term: 1 leader_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } }
I20260929 15:51:33.065330 2674 tablet_replica.cc:442] T 9e741703042847e8845daf7cebedd16b P 9d5c2b08b9854b64a4e1491ca9ae1021: This TxnStatusTablet replica's current role is: LEADER
I20260929 15:51:33.066033 2145 catalog_manager.cc:5495] T 9e741703042847e8845daf7cebedd16b P 9d5c2b08b9854b64a4e1491ca9ae1021 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d5c2b08b9854b64a4e1491ca9ae1021 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } health_report { overall_health: UNKNOWN } } }
I20260929 15:51:33.067637 2676 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 15:51:33.067906 2676 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 15:51:33.084419 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:33.078173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37035 (local address 127.0.0.1:50966)
0929 15:51:33.078398 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:33.078418 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:33.078563 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:33.079038 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:33.079042 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:33.079052 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:51:33.079204 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:33.079211 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:33.080754 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:33.080758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:33.081909 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:33.081921 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:33.082824 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:33.082828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:33.082994 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:33.083047 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:51:33.083954 (+ 907us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:51:33.083974 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 15:51:33.084028 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 15:51:33.084133 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 15:51:33.084507 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:33.078263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50966 (local address 127.0.97.1:37035)
0929 15:51:33.078412 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:33.078420 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:33.078500 (+ 80us) server_negotiation.cc:407] Connection header received
0929 15:51:33.078739 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:33.078745 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:33.078874 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:33.079031 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:51:33.079331 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:33.080593 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:33.082049 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:33.082683 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:33.082822 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:33.083218 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:51:33.083856 (+ 638us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:51:33.084006 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:33.084068 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 15:51:33.084170 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 15:51:33.092000 2443 raft_consensus.cc:1261] T 9e741703042847e8845daf7cebedd16b P 25bedb4d9c1f46aeaa1256eb6708adaa [term 1 FOLLOWER]: Refusing update from remote peer 9d5c2b08b9854b64a4e1491ca9ae1021: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:51:33.092046 2587 raft_consensus.cc:1261] T 9e741703042847e8845daf7cebedd16b P 32d80eed68624cbeb7750d84632e714c [term 1 FOLLOWER]: Refusing update from remote peer 9d5c2b08b9854b64a4e1491ca9ae1021: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:51:33.092928 2674 consensus_queue.cc:1035] T 9e741703042847e8845daf7cebedd16b P 9d5c2b08b9854b64a4e1491ca9ae1021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:51:33.093254 2657 consensus_queue.cc:1035] T 9e741703042847e8845daf7cebedd16b P 9d5c2b08b9854b64a4e1491ca9ae1021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:51:33.104614 2658 tablet_replica.cc:439] T 9e741703042847e8845daf7cebedd16b P 25bedb4d9c1f46aeaa1256eb6708adaa: TxnStatusTablet state changed. Reason: New leader 9d5c2b08b9854b64a4e1491ca9ae1021. Latest consensus state: current_term: 1 leader_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } }
I20260929 15:51:33.104931 2658 tablet_replica.cc:442] T 9e741703042847e8845daf7cebedd16b P 25bedb4d9c1f46aeaa1256eb6708adaa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:51:33.107743 2679 mvcc.cc:205] Tried to move back new op lower bound from 7334695293295828992 to 7334695293184802816. Current Snapshot: MvccSnapshot[applied={T|T < 7334695293295828992}]
I20260929 15:51:33.109613 2659 tablet_replica.cc:439] T 9e741703042847e8845daf7cebedd16b P 32d80eed68624cbeb7750d84632e714c: TxnStatusTablet state changed. Reason: New leader 9d5c2b08b9854b64a4e1491ca9ae1021. Latest consensus state: current_term: 1 leader_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } }
I20260929 15:51:33.109894 2659 tablet_replica.cc:442] T 9e741703042847e8845daf7cebedd16b P 32d80eed68624cbeb7750d84632e714c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:51:33.114230 2674 tablet_replica.cc:439] T 9e741703042847e8845daf7cebedd16b P 9d5c2b08b9854b64a4e1491ca9ae1021: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } }
I20260929 15:51:33.114492 2674 tablet_replica.cc:442] T 9e741703042847e8845daf7cebedd16b P 9d5c2b08b9854b64a4e1491ca9ae1021: This TxnStatusTablet replica's current role is: LEADER
I20260929 15:51:33.116724 2674 tablet_replica.cc:439] T 9e741703042847e8845daf7cebedd16b P 9d5c2b08b9854b64a4e1491ca9ae1021: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } }
I20260929 15:51:33.117007 2674 tablet_replica.cc:442] T 9e741703042847e8845daf7cebedd16b P 9d5c2b08b9854b64a4e1491ca9ae1021: This TxnStatusTablet replica's current role is: LEADER
I20260929 15:51:33.119949 2658 tablet_replica.cc:439] T 9e741703042847e8845daf7cebedd16b P 25bedb4d9c1f46aeaa1256eb6708adaa: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } }
I20260929 15:51:33.120205 2658 tablet_replica.cc:442] T 9e741703042847e8845daf7cebedd16b P 25bedb4d9c1f46aeaa1256eb6708adaa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:51:33.122768 2659 tablet_replica.cc:439] T 9e741703042847e8845daf7cebedd16b P 32d80eed68624cbeb7750d84632e714c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } }
I20260929 15:51:33.123041 2659 tablet_replica.cc:442] T 9e741703042847e8845daf7cebedd16b P 32d80eed68624cbeb7750d84632e714c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:51:33.132984 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:33.126882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59924 (local address 127.0.97.3:43617)
0929 15:51:33.127057 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:33.127065 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:33.127178 (+ 113us) server_negotiation.cc:407] Connection header received
0929 15:51:33.127411 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:33.127414 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:33.127492 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:33.127604 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:51:33.128011 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:33.128701 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:33.130635 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:33.131105 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:33.131197 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:33.131730 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:51:33.132290 (+ 560us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:51:33.132395 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:33.132552 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 15:51:33.132633 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 15:51:33.149124 2582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:33.143947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37035 (local address 127.0.97.3:49847)
0929 15:51:33.144145 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:33.144161 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:33.144256 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:33.144593 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:33.144597 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:33.144608 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:51:33.144762 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:33.144769 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:33.146249 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:33.146253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:33.147262 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:33.147272 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:33.147979 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:33.147984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:33.148113 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:33.148161 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:51:33.148556 (+ 395us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:51:33.148569 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 15:51:33.148662 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 15:51:33.148799 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 15:51:33.149129 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:33.144037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49847 (local address 127.0.97.1:37035)
0929 15:51:33.144242 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:33.144248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:33.144268 (+ 20us) server_negotiation.cc:407] Connection header received
0929 15:51:33.144376 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:33.144381 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:33.144452 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:33.144544 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:51:33.144906 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:33.146110 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:33.147411 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:33.147819 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:33.147968 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:33.148348 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:51:33.148458 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:51:33.148553 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:33.148682 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 15:51:33.148755 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 15:51:33.158569 2443 raft_consensus.cc:1261] T bdd799e6c9ed4123ae947f1086e70ab0 P 25bedb4d9c1f46aeaa1256eb6708adaa [term 1 FOLLOWER]: Refusing update from remote peer 32d80eed68624cbeb7750d84632e714c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:51:33.158979 2309 raft_consensus.cc:1261] T bdd799e6c9ed4123ae947f1086e70ab0 P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 1 FOLLOWER]: Refusing update from remote peer 32d80eed68624cbeb7750d84632e714c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:51:33.159262 2661 consensus_queue.cc:1035] T bdd799e6c9ed4123ae947f1086e70ab0 P 32d80eed68624cbeb7750d84632e714c [LEADER]: Connected to new peer: Peer: permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:51:33.159660 2659 consensus_queue.cc:1035] T bdd799e6c9ed4123ae947f1086e70ab0 P 32d80eed68624cbeb7750d84632e714c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:51:33.165727 2681 mvcc.cc:205] Tried to move back new op lower bound from 7334695293571334144 to 7334695292717629440. Current Snapshot: MvccSnapshot[applied={T|T < 7334695293571334144}]
I20260929 15:51:33.167079 2680 mvcc.cc:205] Tried to move back new op lower bound from 7334695293571334144 to 7334695292717629440. Current Snapshot: MvccSnapshot[applied={T|T < 7334695293571334144}]
I20260929 15:51:33.209272 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:33.194600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53667 (local address 127.0.97.3:43617)
0929 15:51:33.194759 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:33.194765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:33.203755 (+ 8990us) server_negotiation.cc:407] Connection header received
0929 15:51:33.203868 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:33.203876 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:33.203991 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:33.204088 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:51:33.204642 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:33.205756 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:33.207331 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:33.207998 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:33.208125 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:33.208549 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:51:33.208656 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:51:33.208752 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:33.208912 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 15:51:33.208986 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 15:51:33.209455 2695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:33.194191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43617 (local address 127.0.97.1:53667)
0929 15:51:33.203642 (+ 9451us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:33.203659 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:33.203771 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:33.204118 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:33.204123 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:33.204135 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:51:33.204488 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:33.204499 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:33.205896 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:33.205900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:33.207183 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:33.207195 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:33.208171 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:33.208180 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:33.208320 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:33.208372 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:51:33.208758 (+ 386us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:51:33.208774 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 15:51:33.208893 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 15:51:33.209029 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9280,"thread_start_us":9094,"threads_started":1}
I20260929 15:51:33.246203 2147 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:51:33.246536 2147 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:51:33.250466 2147 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8087fc375014469989c805e973593c1b: Sending DeleteTablet for 3 replicas of tablet bdd799e6c9ed4123ae947f1086e70ab0
I20260929 15:51:33.251498 2284 tablet_service.cc:1514] Processing DeleteTablet for tablet bdd799e6c9ed4123ae947f1086e70ab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:51:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50954
I20260929 15:51:33.251655 2423 tablet_service.cc:1514] Processing DeleteTablet for tablet bdd799e6c9ed4123ae947f1086e70ab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:51:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54660
I20260929 15:51:33.252905 2562 tablet_service.cc:1514] Processing DeleteTablet for tablet bdd799e6c9ed4123ae947f1086e70ab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:51:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59922
I20260929 15:51:33.259904 2147 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:51:33.260707 2147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:51:33.270650 2283 tablet_service.cc:1467] Processing CreateTablet for tablet 32fe98e83f804f0e909d7b2bb23de86c (DEFAULT_TABLE table=test-table [id=18f815d24e614f1f9c940ba79f4a44c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:51:33.270713 2423 tablet_service.cc:1467] Processing CreateTablet for tablet 32fe98e83f804f0e909d7b2bb23de86c (DEFAULT_TABLE table=test-table [id=18f815d24e614f1f9c940ba79f4a44c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:51:33.270730 2561 tablet_service.cc:1467] Processing CreateTablet for tablet 32fe98e83f804f0e909d7b2bb23de86c (DEFAULT_TABLE table=test-table [id=18f815d24e614f1f9c940ba79f4a44c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:51:33.271145 2561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32fe98e83f804f0e909d7b2bb23de86c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:33.271160 2423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32fe98e83f804f0e909d7b2bb23de86c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:33.271188 2283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32fe98e83f804f0e909d7b2bb23de86c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:33.271646 2700 tablet_replica.cc:330] T bdd799e6c9ed4123ae947f1086e70ab0 P 25bedb4d9c1f46aeaa1256eb6708adaa: stopping tablet replica
I20260929 15:51:33.273813 2700 raft_consensus.cc:2229] T bdd799e6c9ed4123ae947f1086e70ab0 P 25bedb4d9c1f46aeaa1256eb6708adaa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:51:33.274407 2701 tablet_replica.cc:330] T bdd799e6c9ed4123ae947f1086e70ab0 P 32d80eed68624cbeb7750d84632e714c: stopping tablet replica
I20260929 15:51:33.276202 2700 raft_consensus.cc:2258] T bdd799e6c9ed4123ae947f1086e70ab0 P 25bedb4d9c1f46aeaa1256eb6708adaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:51:33.276715 2701 raft_consensus.cc:2229] T bdd799e6c9ed4123ae947f1086e70ab0 P 32d80eed68624cbeb7750d84632e714c [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:51:33.277532 2653 tablet_bootstrap.cc:492] T 32fe98e83f804f0e909d7b2bb23de86c P 32d80eed68624cbeb7750d84632e714c: Bootstrap starting.
I20260929 15:51:33.278527 2652 tablet_bootstrap.cc:492] T 32fe98e83f804f0e909d7b2bb23de86c P 25bedb4d9c1f46aeaa1256eb6708adaa: Bootstrap starting.
I20260929 15:51:33.279255 2653 tablet_bootstrap.cc:654] T 32fe98e83f804f0e909d7b2bb23de86c P 32d80eed68624cbeb7750d84632e714c: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:33.279723 2701 raft_consensus.cc:2258] T bdd799e6c9ed4123ae947f1086e70ab0 P 32d80eed68624cbeb7750d84632e714c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:51:33.280109 2652 tablet_bootstrap.cc:654] T 32fe98e83f804f0e909d7b2bb23de86c P 25bedb4d9c1f46aeaa1256eb6708adaa: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:33.280839 2699 tablet_replica.cc:330] T bdd799e6c9ed4123ae947f1086e70ab0 P 9d5c2b08b9854b64a4e1491ca9ae1021: stopping tablet replica
I20260929 15:51:33.282145 2651 tablet_bootstrap.cc:492] T 32fe98e83f804f0e909d7b2bb23de86c P 9d5c2b08b9854b64a4e1491ca9ae1021: Bootstrap starting.
I20260929 15:51:33.285619 2651 tablet_bootstrap.cc:654] T 32fe98e83f804f0e909d7b2bb23de86c P 9d5c2b08b9854b64a4e1491ca9ae1021: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:33.290978 2699 raft_consensus.cc:2229] T bdd799e6c9ed4123ae947f1086e70ab0 P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:51:33.292641 2699 raft_consensus.cc:2258] T bdd799e6c9ed4123ae947f1086e70ab0 P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:51:33.295421 2699 ts_tablet_manager.cc:1905] T bdd799e6c9ed4123ae947f1086e70ab0 P 9d5c2b08b9854b64a4e1491ca9ae1021: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:51:33.295955 2652 tablet_bootstrap.cc:492] T 32fe98e83f804f0e909d7b2bb23de86c P 25bedb4d9c1f46aeaa1256eb6708adaa: No bootstrap required, opened a new log
I20260929 15:51:33.296123 2652 ts_tablet_manager.cc:1397] T 32fe98e83f804f0e909d7b2bb23de86c P 25bedb4d9c1f46aeaa1256eb6708adaa: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.008s
I20260929 15:51:33.296904 2652 raft_consensus.cc:348] T 32fe98e83f804f0e909d7b2bb23de86c P 25bedb4d9c1f46aeaa1256eb6708adaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } }
I20260929 15:51:33.297144 2652 raft_consensus.cc:374] T 32fe98e83f804f0e909d7b2bb23de86c P 25bedb4d9c1f46aeaa1256eb6708adaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:33.297211 2652 raft_consensus.cc:729] T 32fe98e83f804f0e909d7b2bb23de86c P 25bedb4d9c1f46aeaa1256eb6708adaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25bedb4d9c1f46aeaa1256eb6708adaa, State: Initialized, Role: FOLLOWER
I20260929 15:51:33.297510 2652 consensus_queue.cc:260] T 32fe98e83f804f0e909d7b2bb23de86c P 25bedb4d9c1f46aeaa1256eb6708adaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } }
I20260929 15:51:33.297716 2701 ts_tablet_manager.cc:1905] T bdd799e6c9ed4123ae947f1086e70ab0 P 32d80eed68624cbeb7750d84632e714c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:51:33.298062 2652 ts_tablet_manager.cc:1428] T 32fe98e83f804f0e909d7b2bb23de86c P 25bedb4d9c1f46aeaa1256eb6708adaa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 15:51:33.299752 2653 tablet_bootstrap.cc:492] T 32fe98e83f804f0e909d7b2bb23de86c P 32d80eed68624cbeb7750d84632e714c: No bootstrap required, opened a new log
I20260929 15:51:33.299909 2653 ts_tablet_manager.cc:1397] T 32fe98e83f804f0e909d7b2bb23de86c P 32d80eed68624cbeb7750d84632e714c: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.000s
I20260929 15:51:33.300680 2653 raft_consensus.cc:348] T 32fe98e83f804f0e909d7b2bb23de86c P 32d80eed68624cbeb7750d84632e714c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } }
I20260929 15:51:33.300904 2653 raft_consensus.cc:374] T 32fe98e83f804f0e909d7b2bb23de86c P 32d80eed68624cbeb7750d84632e714c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:33.300971 2653 raft_consensus.cc:729] T 32fe98e83f804f0e909d7b2bb23de86c P 32d80eed68624cbeb7750d84632e714c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32d80eed68624cbeb7750d84632e714c, State: Initialized, Role: FOLLOWER
I20260929 15:51:33.301182 2653 consensus_queue.cc:260] T 32fe98e83f804f0e909d7b2bb23de86c P 32d80eed68624cbeb7750d84632e714c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } }
I20260929 15:51:33.301782 2653 ts_tablet_manager.cc:1428] T 32fe98e83f804f0e909d7b2bb23de86c P 32d80eed68624cbeb7750d84632e714c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 15:51:33.306058 2700 ts_tablet_manager.cc:1905] T bdd799e6c9ed4123ae947f1086e70ab0 P 25bedb4d9c1f46aeaa1256eb6708adaa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:51:33.312150 2651 tablet_bootstrap.cc:492] T 32fe98e83f804f0e909d7b2bb23de86c P 9d5c2b08b9854b64a4e1491ca9ae1021: No bootstrap required, opened a new log
I20260929 15:51:33.312348 2651 ts_tablet_manager.cc:1397] T 32fe98e83f804f0e909d7b2bb23de86c P 9d5c2b08b9854b64a4e1491ca9ae1021: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.003s
I20260929 15:51:33.313185 2651 raft_consensus.cc:348] T 32fe98e83f804f0e909d7b2bb23de86c P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } }
I20260929 15:51:33.313464 2651 raft_consensus.cc:374] T 32fe98e83f804f0e909d7b2bb23de86c P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:33.313557 2651 raft_consensus.cc:729] T 32fe98e83f804f0e909d7b2bb23de86c P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d5c2b08b9854b64a4e1491ca9ae1021, State: Initialized, Role: FOLLOWER
I20260929 15:51:33.313841 2651 consensus_queue.cc:260] T 32fe98e83f804f0e909d7b2bb23de86c P 9d5c2b08b9854b64a4e1491ca9ae1021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } }
I20260929 15:51:33.316669 2651 ts_tablet_manager.cc:1428] T 32fe98e83f804f0e909d7b2bb23de86c P 9d5c2b08b9854b64a4e1491ca9ae1021: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260929 15:51:33.333212 2701 ts_tablet_manager.cc:1918] T bdd799e6c9ed4123ae947f1086e70ab0 P 32d80eed68624cbeb7750d84632e714c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:51:33.333436 2701 log.cc:1196] T bdd799e6c9ed4123ae947f1086e70ab0 P 32d80eed68624cbeb7750d84632e714c: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-2/wal/wals/bdd799e6c9ed4123ae947f1086e70ab0
I20260929 15:51:33.334057 2701 ts_tablet_manager.cc:1939] T bdd799e6c9ed4123ae947f1086e70ab0 P 32d80eed68624cbeb7750d84632e714c: Deleting consensus metadata
I20260929 15:51:33.335960 2134 catalog_manager.cc:4841] TS 32d80eed68624cbeb7750d84632e714c (127.0.97.3:43617): tablet bdd799e6c9ed4123ae947f1086e70ab0 (table test-table [id=4eeb78f3898d4974822d8f4d689ba4a0]) successfully deleted
I20260929 15:51:33.336241 2700 ts_tablet_manager.cc:1918] T bdd799e6c9ed4123ae947f1086e70ab0 P 25bedb4d9c1f46aeaa1256eb6708adaa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:51:33.336402 2700 log.cc:1196] T bdd799e6c9ed4123ae947f1086e70ab0 P 25bedb4d9c1f46aeaa1256eb6708adaa: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-1/wal/wals/bdd799e6c9ed4123ae947f1086e70ab0
I20260929 15:51:33.336925 2700 ts_tablet_manager.cc:1939] T bdd799e6c9ed4123ae947f1086e70ab0 P 25bedb4d9c1f46aeaa1256eb6708adaa: Deleting consensus metadata
I20260929 15:51:33.337704 2699 ts_tablet_manager.cc:1918] T bdd799e6c9ed4123ae947f1086e70ab0 P 9d5c2b08b9854b64a4e1491ca9ae1021: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:51:33.337934 2699 log.cc:1196] T bdd799e6c9ed4123ae947f1086e70ab0 P 9d5c2b08b9854b64a4e1491ca9ae1021: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-0/wal/wals/bdd799e6c9ed4123ae947f1086e70ab0
I20260929 15:51:33.338696 2132 catalog_manager.cc:4841] TS 25bedb4d9c1f46aeaa1256eb6708adaa (127.0.97.2:33143): tablet bdd799e6c9ed4123ae947f1086e70ab0 (table test-table [id=4eeb78f3898d4974822d8f4d689ba4a0]) successfully deleted
I20260929 15:51:33.338999 2699 ts_tablet_manager.cc:1939] T bdd799e6c9ed4123ae947f1086e70ab0 P 9d5c2b08b9854b64a4e1491ca9ae1021: Deleting consensus metadata
I20260929 15:51:33.340685 2131 catalog_manager.cc:4841] TS 9d5c2b08b9854b64a4e1491ca9ae1021 (127.0.97.1:37035): tablet bdd799e6c9ed4123ae947f1086e70ab0 (table test-table [id=4eeb78f3898d4974822d8f4d689ba4a0]) successfully deleted
I20260929 15:51:33.599659 2658 raft_consensus.cc:482] T 32fe98e83f804f0e909d7b2bb23de86c P 25bedb4d9c1f46aeaa1256eb6708adaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:51:33.599925 2658 raft_consensus.cc:504] T 32fe98e83f804f0e909d7b2bb23de86c P 25bedb4d9c1f46aeaa1256eb6708adaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } }
I20260929 15:51:33.601503 2658 leader_election.cc:288] T 32fe98e83f804f0e909d7b2bb23de86c P 25bedb4d9c1f46aeaa1256eb6708adaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d5c2b08b9854b64a4e1491ca9ae1021 (127.0.97.1:37035), 32d80eed68624cbeb7750d84632e714c (127.0.97.3:43617)
I20260929 15:51:33.608637 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:33.601942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52093 (local address 127.0.97.1:37035)
0929 15:51:33.602129 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:33.602136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:33.602227 (+ 91us) server_negotiation.cc:407] Connection header received
0929 15:51:33.602543 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:33.602549 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:33.602629 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:33.602704 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:51:33.603361 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:33.604429 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:33.606321 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:33.607226 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:33.607350 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:33.607446 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:33.607808 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 15:51:33.607956 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 15:51:33.609419 2705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:33.601671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37035 (local address 127.0.97.2:52093)
0929 15:51:33.602129 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:33.602150 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:33.602353 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:33.602917 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:33.602920 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:33.602940 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:51:33.603153 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:33.603163 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:33.604577 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:33.604583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:33.606169 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:33.606181 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:33.607380 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:33.607386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:33.607577 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:33.607612 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 15:51:33.608839 (+ 1227us) client_negotiation.cc:240] Negotiation successful
0929 15:51:33.608982 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":74,"thread_start_us":470,"threads_started":1}
I20260929 15:51:33.610088 2706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:33.601824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43617 (local address 127.0.97.2:50961)
0929 15:51:33.602649 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:33.602667 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:33.602777 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:33.603123 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:33.603129 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:33.603153 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:51:33.603504 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:33.603512 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:33.604716 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:33.604725 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:33.607063 (+ 2338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:33.607078 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:33.609074 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:33.609080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:33.609250 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:33.609272 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 15:51:33.609418 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 15:51:33.609738 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"mutex_wait_us":43,"thread_start_us":604,"threads_started":1}
I20260929 15:51:33.610066 2309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fe98e83f804f0e909d7b2bb23de86c" candidate_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" is_pre_election: true
I20260929 15:51:33.610333 2309 raft_consensus.cc:2454] T 32fe98e83f804f0e909d7b2bb23de86c P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25bedb4d9c1f46aeaa1256eb6708adaa in term 0.
I20260929 15:51:33.610776 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:33.602226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50961 (local address 127.0.97.3:43617)
0929 15:51:33.602427 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:33.602435 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:33.602756 (+ 321us) server_negotiation.cc:407] Connection header received
0929 15:51:33.602893 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:33.602899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:33.602980 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:33.603098 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:51:33.603648 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:33.604574 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:33.607253 (+ 2679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:33.608627 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:33.609079 (+ 452us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:33.609176 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:33.609439 (+ 263us) server_negotiation.cc:299] Negotiation successful
0929 15:51:33.609738 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 15:51:33.610824 2374 leader_election.cc:302] T 32fe98e83f804f0e909d7b2bb23de86c P 25bedb4d9c1f46aeaa1256eb6708adaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25bedb4d9c1f46aeaa1256eb6708adaa, 9d5c2b08b9854b64a4e1491ca9ae1021; no voters:
I20260929 15:51:33.611194 2658 raft_consensus.cc:2781] T 32fe98e83f804f0e909d7b2bb23de86c P 25bedb4d9c1f46aeaa1256eb6708adaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:51:33.611289 2658 raft_consensus.cc:482] T 32fe98e83f804f0e909d7b2bb23de86c P 25bedb4d9c1f46aeaa1256eb6708adaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:51:33.611346 2658 raft_consensus.cc:3037] T 32fe98e83f804f0e909d7b2bb23de86c P 25bedb4d9c1f46aeaa1256eb6708adaa [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:33.611380 2587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fe98e83f804f0e909d7b2bb23de86c" candidate_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32d80eed68624cbeb7750d84632e714c" is_pre_election: true
I20260929 15:51:33.611606 2587 raft_consensus.cc:2454] T 32fe98e83f804f0e909d7b2bb23de86c P 32d80eed68624cbeb7750d84632e714c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25bedb4d9c1f46aeaa1256eb6708adaa in term 0.
I20260929 15:51:33.613495 2658 raft_consensus.cc:504] T 32fe98e83f804f0e909d7b2bb23de86c P 25bedb4d9c1f46aeaa1256eb6708adaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } }
I20260929 15:51:33.614046 2658 leader_election.cc:288] T 32fe98e83f804f0e909d7b2bb23de86c P 25bedb4d9c1f46aeaa1256eb6708adaa [CANDIDATE]: Term 1 election: Requested vote from peers 9d5c2b08b9854b64a4e1491ca9ae1021 (127.0.97.1:37035), 32d80eed68624cbeb7750d84632e714c (127.0.97.3:43617)
I20260929 15:51:33.614467 2309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fe98e83f804f0e909d7b2bb23de86c" candidate_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021"
I20260929 15:51:33.614565 2587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fe98e83f804f0e909d7b2bb23de86c" candidate_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32d80eed68624cbeb7750d84632e714c"
I20260929 15:51:33.614693 2309 raft_consensus.cc:3037] T 32fe98e83f804f0e909d7b2bb23de86c P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:33.614743 2587 raft_consensus.cc:3037] T 32fe98e83f804f0e909d7b2bb23de86c P 32d80eed68624cbeb7750d84632e714c [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:33.618034 2309 raft_consensus.cc:2454] T 32fe98e83f804f0e909d7b2bb23de86c P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25bedb4d9c1f46aeaa1256eb6708adaa in term 1.
I20260929 15:51:33.618232 2587 raft_consensus.cc:2454] T 32fe98e83f804f0e909d7b2bb23de86c P 32d80eed68624cbeb7750d84632e714c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25bedb4d9c1f46aeaa1256eb6708adaa in term 1.
I20260929 15:51:33.618458 2374 leader_election.cc:302] T 32fe98e83f804f0e909d7b2bb23de86c P 25bedb4d9c1f46aeaa1256eb6708adaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25bedb4d9c1f46aeaa1256eb6708adaa, 9d5c2b08b9854b64a4e1491ca9ae1021; no voters:
I20260929 15:51:33.618737 2658 raft_consensus.cc:2781] T 32fe98e83f804f0e909d7b2bb23de86c P 25bedb4d9c1f46aeaa1256eb6708adaa [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:51:33.619253 2658 raft_consensus.cc:686] T 32fe98e83f804f0e909d7b2bb23de86c P 25bedb4d9c1f46aeaa1256eb6708adaa [term 1 LEADER]: Becoming Leader. State: Replica: 25bedb4d9c1f46aeaa1256eb6708adaa, State: Running, Role: LEADER
I20260929 15:51:33.619830 2658 consensus_queue.cc:237] T 32fe98e83f804f0e909d7b2bb23de86c P 25bedb4d9c1f46aeaa1256eb6708adaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } }
I20260929 15:51:33.623440 2145 catalog_manager.cc:5495] T 32fe98e83f804f0e909d7b2bb23de86c P 25bedb4d9c1f46aeaa1256eb6708adaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 25bedb4d9c1f46aeaa1256eb6708adaa (127.0.97.2). New cstate: current_term: 1 leader_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } health_report { overall_health: UNKNOWN } } }
I20260929 15:51:33.663580 2711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:33.656348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54676 (local address 127.0.97.2:33143)
0929 15:51:33.657208 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:33.657216 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:33.657239 (+ 23us) server_negotiation.cc:407] Connection header received
0929 15:51:33.657334 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:33.657365 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:33.657462 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:33.657650 (+ 188us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:51:33.658085 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:33.659271 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:33.660616 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:33.661257 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:33.661392 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:33.661784 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:51:33.662507 (+ 723us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:51:33.662645 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:33.662738 (+ 93us) server_negotiation.cc:299] Negotiation successful
0929 15:51:33.662807 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":717,"thread_start_us":586,"threads_started":1}
I20260929 15:51:33.677958 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:33.672845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45709 (local address 127.0.97.1:37035)
0929 15:51:33.673021 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:33.673025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:33.673532 (+ 507us) server_negotiation.cc:407] Connection header received
0929 15:51:33.673737 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:33.673746 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:33.673806 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:33.673904 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:51:33.674355 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:33.675017 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:33.676284 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:33.676673 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:33.676799 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:33.677210 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:51:33.677304 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:51:33.677438 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:33.677590 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 15:51:33.677658 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 15:51:33.678076 2714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:33.672738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37035 (local address 127.0.97.2:45709)
0929 15:51:33.673440 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:33.673455 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:33.673599 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:33.673917 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:33.673922 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:33.673933 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:51:33.674192 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:33.674199 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:33.675146 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:33.675152 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:33.676140 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:33.676151 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:33.676853 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:33.676859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:33.676992 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:33.677040 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:51:33.677447 (+ 407us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:51:33.677459 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 15:51:33.677557 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 15:51:33.677706 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":384,"threads_started":1}
I20260929 15:51:33.686484 2309 raft_consensus.cc:1261] T 32fe98e83f804f0e909d7b2bb23de86c P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 1 FOLLOWER]: Refusing update from remote peer 25bedb4d9c1f46aeaa1256eb6708adaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:51:33.687034 2586 raft_consensus.cc:1261] T 32fe98e83f804f0e909d7b2bb23de86c P 32d80eed68624cbeb7750d84632e714c [term 1 FOLLOWER]: Refusing update from remote peer 25bedb4d9c1f46aeaa1256eb6708adaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:51:33.687050 2707 consensus_queue.cc:1035] T 32fe98e83f804f0e909d7b2bb23de86c P 25bedb4d9c1f46aeaa1256eb6708adaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:51:33.687536 2658 consensus_queue.cc:1035] T 32fe98e83f804f0e909d7b2bb23de86c P 25bedb4d9c1f46aeaa1256eb6708adaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:51:33.717703 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:33.711746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33143 (local address 127.0.97.1:34955)
0929 15:51:33.712190 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:33.712210 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:33.712320 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:33.712862 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:33.712868 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:33.712881 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:51:33.713098 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:33.713109 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:33.714504 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:33.714511 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:33.715575 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:33.715591 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:33.716395 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:33.716402 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:33.716547 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:33.716582 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:51:33.717070 (+ 488us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:51:33.717082 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 15:51:33.717159 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 15:51:33.717283 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":380,"threads_started":1}
I20260929 15:51:33.718335 2711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:33.712092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34955 (local address 127.0.97.2:33143)
0929 15:51:33.712258 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:33.712266 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:33.712291 (+ 25us) server_negotiation.cc:407] Connection header received
0929 15:51:33.712464 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:33.712469 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:33.712541 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:33.712612 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:51:33.713256 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:33.714331 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:33.715734 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:33.716231 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:33.716363 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:33.716801 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:51:33.716939 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:51:33.717021 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:33.717981 (+ 960us) server_negotiation.cc:299] Negotiation successful
0929 15:51:33.718060 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260929 15:51:33.719198 2680 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 15:51:35.313292 2147 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:51:35.313606 2147 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:51:35.317588 2147 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8087fc375014469989c805e973593c1b: Sending DeleteTablet for 3 replicas of tablet 32fe98e83f804f0e909d7b2bb23de86c
I20260929 15:51:35.318430 2423 tablet_service.cc:1514] Processing DeleteTablet for tablet 32fe98e83f804f0e909d7b2bb23de86c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:51:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54660
I20260929 15:51:35.318451 2283 tablet_service.cc:1514] Processing DeleteTablet for tablet 32fe98e83f804f0e909d7b2bb23de86c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:51:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50954
I20260929 15:51:35.318529 2562 tablet_service.cc:1514] Processing DeleteTablet for tablet 32fe98e83f804f0e909d7b2bb23de86c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:51:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59922
I20260929 15:51:35.330773 2743 tablet_replica.cc:330] T 32fe98e83f804f0e909d7b2bb23de86c P 25bedb4d9c1f46aeaa1256eb6708adaa: stopping tablet replica
I20260929 15:51:35.331032 2743 raft_consensus.cc:2229] T 32fe98e83f804f0e909d7b2bb23de86c P 25bedb4d9c1f46aeaa1256eb6708adaa [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:51:35.331606 2743 raft_consensus.cc:2258] T 32fe98e83f804f0e909d7b2bb23de86c P 25bedb4d9c1f46aeaa1256eb6708adaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:51:35.332952 2744 tablet_replica.cc:330] T 32fe98e83f804f0e909d7b2bb23de86c P 32d80eed68624cbeb7750d84632e714c: stopping tablet replica
I20260929 15:51:35.333153 2744 raft_consensus.cc:2229] T 32fe98e83f804f0e909d7b2bb23de86c P 32d80eed68624cbeb7750d84632e714c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:51:35.333451 2742 tablet_replica.cc:330] T 32fe98e83f804f0e909d7b2bb23de86c P 9d5c2b08b9854b64a4e1491ca9ae1021: stopping tablet replica
I20260929 15:51:35.333436 2744 raft_consensus.cc:2258] T 32fe98e83f804f0e909d7b2bb23de86c P 32d80eed68624cbeb7750d84632e714c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:51:35.333598 2743 ts_tablet_manager.cc:1905] T 32fe98e83f804f0e909d7b2bb23de86c P 25bedb4d9c1f46aeaa1256eb6708adaa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:51:35.333660 2742 raft_consensus.cc:2229] T 32fe98e83f804f0e909d7b2bb23de86c P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:51:35.333917 2742 raft_consensus.cc:2258] T 32fe98e83f804f0e909d7b2bb23de86c P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:51:35.334424 2744 ts_tablet_manager.cc:1905] T 32fe98e83f804f0e909d7b2bb23de86c P 32d80eed68624cbeb7750d84632e714c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:51:35.334837 2742 ts_tablet_manager.cc:1905] T 32fe98e83f804f0e909d7b2bb23de86c P 9d5c2b08b9854b64a4e1491ca9ae1021: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:51:35.338857 2743 ts_tablet_manager.cc:1918] T 32fe98e83f804f0e909d7b2bb23de86c P 25bedb4d9c1f46aeaa1256eb6708adaa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 15:51:35.339003 2743 log.cc:1196] T 32fe98e83f804f0e909d7b2bb23de86c P 25bedb4d9c1f46aeaa1256eb6708adaa: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-1/wal/wals/32fe98e83f804f0e909d7b2bb23de86c
I20260929 15:51:35.339370 2743 ts_tablet_manager.cc:1939] T 32fe98e83f804f0e909d7b2bb23de86c P 25bedb4d9c1f46aeaa1256eb6708adaa: Deleting consensus metadata
I20260929 15:51:35.340452 2132 catalog_manager.cc:4841] TS 25bedb4d9c1f46aeaa1256eb6708adaa (127.0.97.2:33143): tablet 32fe98e83f804f0e909d7b2bb23de86c (table test-table [id=18f815d24e614f1f9c940ba79f4a44c9]) successfully deleted
I20260929 15:51:35.341713 2744 ts_tablet_manager.cc:1918] T 32fe98e83f804f0e909d7b2bb23de86c P 32d80eed68624cbeb7750d84632e714c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 15:51:35.341810 2742 ts_tablet_manager.cc:1918] T 32fe98e83f804f0e909d7b2bb23de86c P 9d5c2b08b9854b64a4e1491ca9ae1021: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 15:51:35.341871 2744 log.cc:1196] T 32fe98e83f804f0e909d7b2bb23de86c P 32d80eed68624cbeb7750d84632e714c: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-2/wal/wals/32fe98e83f804f0e909d7b2bb23de86c
I20260929 15:51:35.341969 2742 log.cc:1196] T 32fe98e83f804f0e909d7b2bb23de86c P 9d5c2b08b9854b64a4e1491ca9ae1021: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-0/wal/wals/32fe98e83f804f0e909d7b2bb23de86c
I20260929 15:51:35.342386 2744 ts_tablet_manager.cc:1939] T 32fe98e83f804f0e909d7b2bb23de86c P 32d80eed68624cbeb7750d84632e714c: Deleting consensus metadata
I20260929 15:51:35.342397 2742 ts_tablet_manager.cc:1939] T 32fe98e83f804f0e909d7b2bb23de86c P 9d5c2b08b9854b64a4e1491ca9ae1021: Deleting consensus metadata
I20260929 15:51:35.343451 2131 catalog_manager.cc:4841] TS 9d5c2b08b9854b64a4e1491ca9ae1021 (127.0.97.1:37035): tablet 32fe98e83f804f0e909d7b2bb23de86c (table test-table [id=18f815d24e614f1f9c940ba79f4a44c9]) successfully deleted
I20260929 15:51:35.343407 2134 catalog_manager.cc:4841] TS 32d80eed68624cbeb7750d84632e714c (127.0.97.3:43617): tablet 32fe98e83f804f0e909d7b2bb23de86c (table test-table [id=18f815d24e614f1f9c940ba79f4a44c9]) successfully deleted
Sep 29 15:51:35 dist-test-slave-r0ck krb5kdc[2102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697095, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 15:51:35 dist-test-slave-r0ck krb5kdc[2102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697095, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:51:35.380708 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:35.366440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42862 (local address 127.0.97.62:36055)
0929 15:51:35.367000 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:35.367008 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:35.367051 (+ 43us) server_negotiation.cc:407] Connection header received
0929 15:51:35.367273 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:35.367278 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:35.367339 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:35.367451 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:35.368850 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:35.369720 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:35.370971 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:35.371351 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:35.371513 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:35.375504 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:35.375509 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:35.375512 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:35.375543 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:35.378153 (+ 2610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:35.378808 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:35.378813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:35.378817 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:35.378949 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:35.379311 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:35.379322 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:35.379325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:35.379599 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:35.379834 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:35.380171 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 15:51:35.380390 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":272,"threads_started":1}
I20260929 15:51:35.384150 2147 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:51:35.384708 2147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:51:35.391232 2283 tablet_service.cc:1467] Processing CreateTablet for tablet c049be280394447899ad8b003370d406 (DEFAULT_TABLE table=test-table [id=b01cdf60392e4b44bb1dd8c96f66c6a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:51:35.391700 2283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c049be280394447899ad8b003370d406. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:35.391675 2423 tablet_service.cc:1467] Processing CreateTablet for tablet c049be280394447899ad8b003370d406 (DEFAULT_TABLE table=test-table [id=b01cdf60392e4b44bb1dd8c96f66c6a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:51:35.392117 2423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c049be280394447899ad8b003370d406. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:35.396876 2753 tablet_bootstrap.cc:492] T c049be280394447899ad8b003370d406 P 9d5c2b08b9854b64a4e1491ca9ae1021: Bootstrap starting.
I20260929 15:51:35.397423 2562 tablet_service.cc:1467] Processing CreateTablet for tablet c049be280394447899ad8b003370d406 (DEFAULT_TABLE table=test-table [id=b01cdf60392e4b44bb1dd8c96f66c6a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:51:35.397547 2754 tablet_bootstrap.cc:492] T c049be280394447899ad8b003370d406 P 25bedb4d9c1f46aeaa1256eb6708adaa: Bootstrap starting.
I20260929 15:51:35.397817 2562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c049be280394447899ad8b003370d406. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:35.398615 2753 tablet_bootstrap.cc:654] T c049be280394447899ad8b003370d406 P 9d5c2b08b9854b64a4e1491ca9ae1021: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:35.399221 2754 tablet_bootstrap.cc:654] T c049be280394447899ad8b003370d406 P 25bedb4d9c1f46aeaa1256eb6708adaa: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:35.400995 2753 tablet_bootstrap.cc:492] T c049be280394447899ad8b003370d406 P 9d5c2b08b9854b64a4e1491ca9ae1021: No bootstrap required, opened a new log
I20260929 15:51:35.401168 2753 ts_tablet_manager.cc:1397] T c049be280394447899ad8b003370d406 P 9d5c2b08b9854b64a4e1491ca9ae1021: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 15:51:35.402176 2753 raft_consensus.cc:348] T c049be280394447899ad8b003370d406 P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } }
I20260929 15:51:35.402546 2753 raft_consensus.cc:374] T c049be280394447899ad8b003370d406 P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:35.402659 2753 raft_consensus.cc:729] T c049be280394447899ad8b003370d406 P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d5c2b08b9854b64a4e1491ca9ae1021, State: Initialized, Role: FOLLOWER
I20260929 15:51:35.402961 2753 consensus_queue.cc:260] T c049be280394447899ad8b003370d406 P 9d5c2b08b9854b64a4e1491ca9ae1021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } }
I20260929 15:51:35.403721 2753 ts_tablet_manager.cc:1428] T c049be280394447899ad8b003370d406 P 9d5c2b08b9854b64a4e1491ca9ae1021: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260929 15:51:35.405118 2754 tablet_bootstrap.cc:492] T c049be280394447899ad8b003370d406 P 25bedb4d9c1f46aeaa1256eb6708adaa: No bootstrap required, opened a new log
I20260929 15:51:35.405253 2754 ts_tablet_manager.cc:1397] T c049be280394447899ad8b003370d406 P 25bedb4d9c1f46aeaa1256eb6708adaa: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260929 15:51:35.406038 2754 raft_consensus.cc:348] T c049be280394447899ad8b003370d406 P 25bedb4d9c1f46aeaa1256eb6708adaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } }
I20260929 15:51:35.406229 2754 raft_consensus.cc:374] T c049be280394447899ad8b003370d406 P 25bedb4d9c1f46aeaa1256eb6708adaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:35.406402 2754 raft_consensus.cc:729] T c049be280394447899ad8b003370d406 P 25bedb4d9c1f46aeaa1256eb6708adaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25bedb4d9c1f46aeaa1256eb6708adaa, State: Initialized, Role: FOLLOWER
I20260929 15:51:35.407075 2754 consensus_queue.cc:260] T c049be280394447899ad8b003370d406 P 25bedb4d9c1f46aeaa1256eb6708adaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } }
I20260929 15:51:35.407732 2754 ts_tablet_manager.cc:1428] T c049be280394447899ad8b003370d406 P 25bedb4d9c1f46aeaa1256eb6708adaa: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260929 15:51:35.409824 2757 tablet_bootstrap.cc:492] T c049be280394447899ad8b003370d406 P 32d80eed68624cbeb7750d84632e714c: Bootstrap starting.
I20260929 15:51:35.411545 2757 tablet_bootstrap.cc:654] T c049be280394447899ad8b003370d406 P 32d80eed68624cbeb7750d84632e714c: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:35.413478 2757 tablet_bootstrap.cc:492] T c049be280394447899ad8b003370d406 P 32d80eed68624cbeb7750d84632e714c: No bootstrap required, opened a new log
I20260929 15:51:35.413636 2757 ts_tablet_manager.cc:1397] T c049be280394447899ad8b003370d406 P 32d80eed68624cbeb7750d84632e714c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 15:51:35.414206 2757 raft_consensus.cc:348] T c049be280394447899ad8b003370d406 P 32d80eed68624cbeb7750d84632e714c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } }
I20260929 15:51:35.414407 2757 raft_consensus.cc:374] T c049be280394447899ad8b003370d406 P 32d80eed68624cbeb7750d84632e714c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:35.414496 2757 raft_consensus.cc:729] T c049be280394447899ad8b003370d406 P 32d80eed68624cbeb7750d84632e714c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32d80eed68624cbeb7750d84632e714c, State: Initialized, Role: FOLLOWER
I20260929 15:51:35.414736 2757 consensus_queue.cc:260] T c049be280394447899ad8b003370d406 P 32d80eed68624cbeb7750d84632e714c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } }
I20260929 15:51:35.415446 2757 ts_tablet_manager.cc:1428] T c049be280394447899ad8b003370d406 P 32d80eed68624cbeb7750d84632e714c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 15:51:35.448266 2723 raft_consensus.cc:482] T c049be280394447899ad8b003370d406 P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:51:35.448436 2723 raft_consensus.cc:504] T c049be280394447899ad8b003370d406 P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } }
I20260929 15:51:35.449168 2723 leader_election.cc:288] T c049be280394447899ad8b003370d406 P 9d5c2b08b9854b64a4e1491ca9ae1021 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25bedb4d9c1f46aeaa1256eb6708adaa (127.0.97.2:33143), 32d80eed68624cbeb7750d84632e714c (127.0.97.3:43617)
I20260929 15:51:35.449705 2442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c049be280394447899ad8b003370d406" candidate_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" is_pre_election: true
I20260929 15:51:35.449736 2587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c049be280394447899ad8b003370d406" candidate_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32d80eed68624cbeb7750d84632e714c" is_pre_election: true
I20260929 15:51:35.449939 2442 raft_consensus.cc:2454] T c049be280394447899ad8b003370d406 P 25bedb4d9c1f46aeaa1256eb6708adaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d5c2b08b9854b64a4e1491ca9ae1021 in term 0.
I20260929 15:51:35.449991 2587 raft_consensus.cc:2454] T c049be280394447899ad8b003370d406 P 32d80eed68624cbeb7750d84632e714c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d5c2b08b9854b64a4e1491ca9ae1021 in term 0.
I20260929 15:51:35.450471 2235 leader_election.cc:302] T c049be280394447899ad8b003370d406 P 9d5c2b08b9854b64a4e1491ca9ae1021 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 25bedb4d9c1f46aeaa1256eb6708adaa, 32d80eed68624cbeb7750d84632e714c, 9d5c2b08b9854b64a4e1491ca9ae1021; no voters:
I20260929 15:51:35.450789 2723 raft_consensus.cc:2781] T c049be280394447899ad8b003370d406 P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:51:35.450898 2723 raft_consensus.cc:482] T c049be280394447899ad8b003370d406 P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:51:35.451040 2723 raft_consensus.cc:3037] T c049be280394447899ad8b003370d406 P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:35.453037 2723 raft_consensus.cc:504] T c049be280394447899ad8b003370d406 P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } }
I20260929 15:51:35.453672 2723 leader_election.cc:288] T c049be280394447899ad8b003370d406 P 9d5c2b08b9854b64a4e1491ca9ae1021 [CANDIDATE]: Term 1 election: Requested vote from peers 25bedb4d9c1f46aeaa1256eb6708adaa (127.0.97.2:33143), 32d80eed68624cbeb7750d84632e714c (127.0.97.3:43617)
I20260929 15:51:35.454192 2442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c049be280394447899ad8b003370d406" candidate_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa"
I20260929 15:51:35.454152 2587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c049be280394447899ad8b003370d406" candidate_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32d80eed68624cbeb7750d84632e714c"
I20260929 15:51:35.454381 2442 raft_consensus.cc:3037] T c049be280394447899ad8b003370d406 P 25bedb4d9c1f46aeaa1256eb6708adaa [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:35.454411 2587 raft_consensus.cc:3037] T c049be280394447899ad8b003370d406 P 32d80eed68624cbeb7750d84632e714c [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:35.457664 2587 raft_consensus.cc:2454] T c049be280394447899ad8b003370d406 P 32d80eed68624cbeb7750d84632e714c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d5c2b08b9854b64a4e1491ca9ae1021 in term 1.
I20260929 15:51:35.457729 2442 raft_consensus.cc:2454] T c049be280394447899ad8b003370d406 P 25bedb4d9c1f46aeaa1256eb6708adaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d5c2b08b9854b64a4e1491ca9ae1021 in term 1.
I20260929 15:51:35.458086 2235 leader_election.cc:302] T c049be280394447899ad8b003370d406 P 9d5c2b08b9854b64a4e1491ca9ae1021 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 25bedb4d9c1f46aeaa1256eb6708adaa, 32d80eed68624cbeb7750d84632e714c, 9d5c2b08b9854b64a4e1491ca9ae1021; no voters:
I20260929 15:51:35.458359 2723 raft_consensus.cc:2781] T c049be280394447899ad8b003370d406 P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:51:35.458516 2723 raft_consensus.cc:686] T c049be280394447899ad8b003370d406 P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 1 LEADER]: Becoming Leader. State: Replica: 9d5c2b08b9854b64a4e1491ca9ae1021, State: Running, Role: LEADER
I20260929 15:51:35.458789 2723 consensus_queue.cc:237] T c049be280394447899ad8b003370d406 P 9d5c2b08b9854b64a4e1491ca9ae1021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } }
I20260929 15:51:35.461383 2147 catalog_manager.cc:5495] T c049be280394447899ad8b003370d406 P 9d5c2b08b9854b64a4e1491ca9ae1021 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d5c2b08b9854b64a4e1491ca9ae1021 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } health_report { overall_health: UNKNOWN } } }
I20260929 15:51:35.489626 2762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:35.484951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50972 (local address 127.0.97.1:37035)
0929 15:51:35.485497 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:35.485501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:35.485520 (+ 19us) server_negotiation.cc:407] Connection header received
0929 15:51:35.485568 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:35.485571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:35.485631 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:35.485712 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:51:35.486064 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:35.486796 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:35.488077 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:35.488467 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:35.488569 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:35.488921 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:51:35.489012 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:51:35.489081 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:35.489194 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 15:51:35.489248 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":260,"threads_started":1}
I20260929 15:51:35.502170 2762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:35.497549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40781 (local address 127.0.97.1:37035)
0929 15:51:35.497687 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:35.497695 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:35.498125 (+ 430us) server_negotiation.cc:407] Connection header received
0929 15:51:35.498253 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:35.498256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:35.498310 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:35.498389 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:51:35.498827 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:35.499496 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:35.500759 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:35.501123 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:35.501213 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:35.501637 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:51:35.501720 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:51:35.501784 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:35.501923 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 15:51:35.501977 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 15:51:35.502321 2765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:35.497314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37035 (local address 127.0.97.1:40781)
0929 15:51:35.498040 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:35.498055 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:35.498161 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:35.498442 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:35.498446 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:35.498456 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:51:35.498698 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:35.498705 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:35.499648 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:35.499653 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:35.500622 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:35.500632 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:35.501274 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:35.501278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:35.501476 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:35.501509 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:51:35.501799 (+ 290us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:51:35.501830 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 15:51:35.501890 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 15:51:35.501988 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"mutex_wait_us":2,"thread_start_us":415,"threads_started":1}
I20260929 15:51:35.509783 2587 raft_consensus.cc:1261] T c049be280394447899ad8b003370d406 P 32d80eed68624cbeb7750d84632e714c [term 1 FOLLOWER]: Refusing update from remote peer 9d5c2b08b9854b64a4e1491ca9ae1021: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:51:35.509778 2442 raft_consensus.cc:1261] T c049be280394447899ad8b003370d406 P 25bedb4d9c1f46aeaa1256eb6708adaa [term 1 FOLLOWER]: Refusing update from remote peer 9d5c2b08b9854b64a4e1491ca9ae1021: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:51:35.510264 2723 consensus_queue.cc:1035] T c049be280394447899ad8b003370d406 P 9d5c2b08b9854b64a4e1491ca9ae1021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:51:35.510555 2686 consensus_queue.cc:1035] T c049be280394447899ad8b003370d406 P 9d5c2b08b9854b64a4e1491ca9ae1021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:51:35.554101 2145 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:51:35.554296 2145 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:51:35.557060 2145 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8087fc375014469989c805e973593c1b: Sending DeleteTablet for 3 replicas of tablet c049be280394447899ad8b003370d406
I20260929 15:51:35.557737 2283 tablet_service.cc:1514] Processing DeleteTablet for tablet c049be280394447899ad8b003370d406 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:51:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50954
I20260929 15:51:35.557893 2423 tablet_service.cc:1514] Processing DeleteTablet for tablet c049be280394447899ad8b003370d406 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:51:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54660
I20260929 15:51:35.557945 2562 tablet_service.cc:1514] Processing DeleteTablet for tablet c049be280394447899ad8b003370d406 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:51:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59922
I20260929 15:51:35.558179 2743 tablet_replica.cc:330] T c049be280394447899ad8b003370d406 P 25bedb4d9c1f46aeaa1256eb6708adaa: stopping tablet replica
I20260929 15:51:35.558215 2742 tablet_replica.cc:330] T c049be280394447899ad8b003370d406 P 9d5c2b08b9854b64a4e1491ca9ae1021: stopping tablet replica
I20260929 15:51:35.558336 2744 tablet_replica.cc:330] T c049be280394447899ad8b003370d406 P 32d80eed68624cbeb7750d84632e714c: stopping tablet replica
I20260929 15:51:35.558393 2742 raft_consensus.cc:2229] T c049be280394447899ad8b003370d406 P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:51:35.558382 2743 raft_consensus.cc:2229] T c049be280394447899ad8b003370d406 P 25bedb4d9c1f46aeaa1256eb6708adaa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:51:35.558540 2744 raft_consensus.cc:2229] T c049be280394447899ad8b003370d406 P 32d80eed68624cbeb7750d84632e714c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:51:35.558554 2743 raft_consensus.cc:2258] T c049be280394447899ad8b003370d406 P 25bedb4d9c1f46aeaa1256eb6708adaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:51:35.558712 2744 raft_consensus.cc:2258] T c049be280394447899ad8b003370d406 P 32d80eed68624cbeb7750d84632e714c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:51:35.558885 2742 raft_consensus.cc:2258] T c049be280394447899ad8b003370d406 P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:51:35.560214 2742 ts_tablet_manager.cc:1905] T c049be280394447899ad8b003370d406 P 9d5c2b08b9854b64a4e1491ca9ae1021: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:51:35.560272 2744 ts_tablet_manager.cc:1905] T c049be280394447899ad8b003370d406 P 32d80eed68624cbeb7750d84632e714c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:51:35.565573 2742 ts_tablet_manager.cc:1918] T c049be280394447899ad8b003370d406 P 9d5c2b08b9854b64a4e1491ca9ae1021: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:51:35.565743 2742 log.cc:1196] T c049be280394447899ad8b003370d406 P 9d5c2b08b9854b64a4e1491ca9ae1021: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-0/wal/wals/c049be280394447899ad8b003370d406
I20260929 15:51:35.566573 2742 ts_tablet_manager.cc:1939] T c049be280394447899ad8b003370d406 P 9d5c2b08b9854b64a4e1491ca9ae1021: Deleting consensus metadata
I20260929 15:51:35.566692 2744 ts_tablet_manager.cc:1918] T c049be280394447899ad8b003370d406 P 32d80eed68624cbeb7750d84632e714c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:51:35.566836 2744 log.cc:1196] T c049be280394447899ad8b003370d406 P 32d80eed68624cbeb7750d84632e714c: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-2/wal/wals/c049be280394447899ad8b003370d406
I20260929 15:51:35.567231 2744 ts_tablet_manager.cc:1939] T c049be280394447899ad8b003370d406 P 32d80eed68624cbeb7750d84632e714c: Deleting consensus metadata
I20260929 15:51:35.567703 2131 catalog_manager.cc:4841] TS 9d5c2b08b9854b64a4e1491ca9ae1021 (127.0.97.1:37035): tablet c049be280394447899ad8b003370d406 (table test-table [id=b01cdf60392e4b44bb1dd8c96f66c6a3]) successfully deleted
I20260929 15:51:35.568205 2134 catalog_manager.cc:4841] TS 32d80eed68624cbeb7750d84632e714c (127.0.97.3:43617): tablet c049be280394447899ad8b003370d406 (table test-table [id=b01cdf60392e4b44bb1dd8c96f66c6a3]) successfully deleted
I20260929 15:51:35.568861 2743 ts_tablet_manager.cc:1905] T c049be280394447899ad8b003370d406 P 25bedb4d9c1f46aeaa1256eb6708adaa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:51:35.568815 2145 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:51:35.569517 2145 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:51:35.575408 2743 ts_tablet_manager.cc:1918] T c049be280394447899ad8b003370d406 P 25bedb4d9c1f46aeaa1256eb6708adaa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:51:35.575567 2743 log.cc:1196] T c049be280394447899ad8b003370d406 P 25bedb4d9c1f46aeaa1256eb6708adaa: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-1/wal/wals/c049be280394447899ad8b003370d406
I20260929 15:51:35.575840 2283 tablet_service.cc:1467] Processing CreateTablet for tablet 67e00dbf892349ffbadfe7527ba92222 (DEFAULT_TABLE table=test-table [id=bb3d829b9c1444058e34285fc25849e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:51:35.575947 2743 ts_tablet_manager.cc:1939] T c049be280394447899ad8b003370d406 P 25bedb4d9c1f46aeaa1256eb6708adaa: Deleting consensus metadata
I20260929 15:51:35.576251 2283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67e00dbf892349ffbadfe7527ba92222. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:35.576169 2562 tablet_service.cc:1467] Processing CreateTablet for tablet 67e00dbf892349ffbadfe7527ba92222 (DEFAULT_TABLE table=test-table [id=bb3d829b9c1444058e34285fc25849e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:51:35.576524 2562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67e00dbf892349ffbadfe7527ba92222. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:35.577179 2132 catalog_manager.cc:4841] TS 25bedb4d9c1f46aeaa1256eb6708adaa (127.0.97.2:33143): tablet c049be280394447899ad8b003370d406 (table test-table [id=b01cdf60392e4b44bb1dd8c96f66c6a3]) successfully deleted
I20260929 15:51:35.577867 2423 tablet_service.cc:1467] Processing CreateTablet for tablet 67e00dbf892349ffbadfe7527ba92222 (DEFAULT_TABLE table=test-table [id=bb3d829b9c1444058e34285fc25849e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:51:35.578210 2423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67e00dbf892349ffbadfe7527ba92222. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:35.583357 2753 tablet_bootstrap.cc:492] T 67e00dbf892349ffbadfe7527ba92222 P 9d5c2b08b9854b64a4e1491ca9ae1021: Bootstrap starting.
I20260929 15:51:35.583849 2757 tablet_bootstrap.cc:492] T 67e00dbf892349ffbadfe7527ba92222 P 32d80eed68624cbeb7750d84632e714c: Bootstrap starting.
I20260929 15:51:35.584650 2754 tablet_bootstrap.cc:492] T 67e00dbf892349ffbadfe7527ba92222 P 25bedb4d9c1f46aeaa1256eb6708adaa: Bootstrap starting.
I20260929 15:51:35.585031 2753 tablet_bootstrap.cc:654] T 67e00dbf892349ffbadfe7527ba92222 P 9d5c2b08b9854b64a4e1491ca9ae1021: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:35.585419 2757 tablet_bootstrap.cc:654] T 67e00dbf892349ffbadfe7527ba92222 P 32d80eed68624cbeb7750d84632e714c: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:35.586243 2754 tablet_bootstrap.cc:654] T 67e00dbf892349ffbadfe7527ba92222 P 25bedb4d9c1f46aeaa1256eb6708adaa: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:35.595388 2753 tablet_bootstrap.cc:492] T 67e00dbf892349ffbadfe7527ba92222 P 9d5c2b08b9854b64a4e1491ca9ae1021: No bootstrap required, opened a new log
I20260929 15:51:35.595679 2753 ts_tablet_manager.cc:1397] T 67e00dbf892349ffbadfe7527ba92222 P 9d5c2b08b9854b64a4e1491ca9ae1021: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.010s
I20260929 15:51:35.596743 2753 raft_consensus.cc:348] T 67e00dbf892349ffbadfe7527ba92222 P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } }
I20260929 15:51:35.597283 2753 raft_consensus.cc:374] T 67e00dbf892349ffbadfe7527ba92222 P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:35.597453 2753 raft_consensus.cc:729] T 67e00dbf892349ffbadfe7527ba92222 P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d5c2b08b9854b64a4e1491ca9ae1021, State: Initialized, Role: FOLLOWER
I20260929 15:51:35.597893 2753 consensus_queue.cc:260] T 67e00dbf892349ffbadfe7527ba92222 P 9d5c2b08b9854b64a4e1491ca9ae1021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } }
I20260929 15:51:35.598726 2753 ts_tablet_manager.cc:1428] T 67e00dbf892349ffbadfe7527ba92222 P 9d5c2b08b9854b64a4e1491ca9ae1021: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 15:51:35.599663 2757 tablet_bootstrap.cc:492] T 67e00dbf892349ffbadfe7527ba92222 P 32d80eed68624cbeb7750d84632e714c: No bootstrap required, opened a new log
I20260929 15:51:35.599826 2757 ts_tablet_manager.cc:1397] T 67e00dbf892349ffbadfe7527ba92222 P 32d80eed68624cbeb7750d84632e714c: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.005s
I20260929 15:51:35.600458 2757 raft_consensus.cc:348] T 67e00dbf892349ffbadfe7527ba92222 P 32d80eed68624cbeb7750d84632e714c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } }
I20260929 15:51:35.600641 2757 raft_consensus.cc:374] T 67e00dbf892349ffbadfe7527ba92222 P 32d80eed68624cbeb7750d84632e714c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:35.600735 2757 raft_consensus.cc:729] T 67e00dbf892349ffbadfe7527ba92222 P 32d80eed68624cbeb7750d84632e714c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32d80eed68624cbeb7750d84632e714c, State: Initialized, Role: FOLLOWER
I20260929 15:51:35.602393 2754 tablet_bootstrap.cc:492] T 67e00dbf892349ffbadfe7527ba92222 P 25bedb4d9c1f46aeaa1256eb6708adaa: No bootstrap required, opened a new log
I20260929 15:51:35.601028 2757 consensus_queue.cc:260] T 67e00dbf892349ffbadfe7527ba92222 P 32d80eed68624cbeb7750d84632e714c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } }
I20260929 15:51:35.602615 2754 ts_tablet_manager.cc:1397] T 67e00dbf892349ffbadfe7527ba92222 P 25bedb4d9c1f46aeaa1256eb6708adaa: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.010s
I20260929 15:51:35.603793 2754 raft_consensus.cc:348] T 67e00dbf892349ffbadfe7527ba92222 P 25bedb4d9c1f46aeaa1256eb6708adaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } }
I20260929 15:51:35.604034 2754 raft_consensus.cc:374] T 67e00dbf892349ffbadfe7527ba92222 P 25bedb4d9c1f46aeaa1256eb6708adaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:35.604107 2754 raft_consensus.cc:729] T 67e00dbf892349ffbadfe7527ba92222 P 25bedb4d9c1f46aeaa1256eb6708adaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25bedb4d9c1f46aeaa1256eb6708adaa, State: Initialized, Role: FOLLOWER
I20260929 15:51:35.604379 2754 consensus_queue.cc:260] T 67e00dbf892349ffbadfe7527ba92222 P 25bedb4d9c1f46aeaa1256eb6708adaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } }
I20260929 15:51:35.605530 2754 ts_tablet_manager.cc:1428] T 67e00dbf892349ffbadfe7527ba92222 P 25bedb4d9c1f46aeaa1256eb6708adaa: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260929 15:51:35.605664 2757 ts_tablet_manager.cc:1428] T 67e00dbf892349ffbadfe7527ba92222 P 32d80eed68624cbeb7750d84632e714c: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260929 15:51:35.624275 2736 raft_consensus.cc:482] T 67e00dbf892349ffbadfe7527ba92222 P 25bedb4d9c1f46aeaa1256eb6708adaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:51:35.624750 2736 raft_consensus.cc:504] T 67e00dbf892349ffbadfe7527ba92222 P 25bedb4d9c1f46aeaa1256eb6708adaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } }
I20260929 15:51:35.625569 2736 leader_election.cc:288] T 67e00dbf892349ffbadfe7527ba92222 P 25bedb4d9c1f46aeaa1256eb6708adaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d5c2b08b9854b64a4e1491ca9ae1021 (127.0.97.1:37035), 32d80eed68624cbeb7750d84632e714c (127.0.97.3:43617)
I20260929 15:51:35.626101 2309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e00dbf892349ffbadfe7527ba92222" candidate_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" is_pre_election: true
I20260929 15:51:35.626159 2586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e00dbf892349ffbadfe7527ba92222" candidate_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32d80eed68624cbeb7750d84632e714c" is_pre_election: true
I20260929 15:51:35.626405 2586 raft_consensus.cc:2454] T 67e00dbf892349ffbadfe7527ba92222 P 32d80eed68624cbeb7750d84632e714c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25bedb4d9c1f46aeaa1256eb6708adaa in term 0.
I20260929 15:51:35.626389 2309 raft_consensus.cc:2454] T 67e00dbf892349ffbadfe7527ba92222 P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25bedb4d9c1f46aeaa1256eb6708adaa in term 0.
I20260929 15:51:35.626904 2377 leader_election.cc:302] T 67e00dbf892349ffbadfe7527ba92222 P 25bedb4d9c1f46aeaa1256eb6708adaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 25bedb4d9c1f46aeaa1256eb6708adaa, 32d80eed68624cbeb7750d84632e714c, 9d5c2b08b9854b64a4e1491ca9ae1021; no voters:
I20260929 15:51:35.627557 2736 raft_consensus.cc:2781] T 67e00dbf892349ffbadfe7527ba92222 P 25bedb4d9c1f46aeaa1256eb6708adaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:51:35.627687 2736 raft_consensus.cc:482] T 67e00dbf892349ffbadfe7527ba92222 P 25bedb4d9c1f46aeaa1256eb6708adaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:51:35.627800 2736 raft_consensus.cc:3037] T 67e00dbf892349ffbadfe7527ba92222 P 25bedb4d9c1f46aeaa1256eb6708adaa [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:35.630203 2736 raft_consensus.cc:504] T 67e00dbf892349ffbadfe7527ba92222 P 25bedb4d9c1f46aeaa1256eb6708adaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } }
I20260929 15:51:35.630822 2736 leader_election.cc:288] T 67e00dbf892349ffbadfe7527ba92222 P 25bedb4d9c1f46aeaa1256eb6708adaa [CANDIDATE]: Term 1 election: Requested vote from peers 9d5c2b08b9854b64a4e1491ca9ae1021 (127.0.97.1:37035), 32d80eed68624cbeb7750d84632e714c (127.0.97.3:43617)
I20260929 15:51:35.631218 2309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e00dbf892349ffbadfe7527ba92222" candidate_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021"
I20260929 15:51:35.631325 2586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e00dbf892349ffbadfe7527ba92222" candidate_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32d80eed68624cbeb7750d84632e714c"
I20260929 15:51:35.631439 2309 raft_consensus.cc:3037] T 67e00dbf892349ffbadfe7527ba92222 P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:35.631587 2586 raft_consensus.cc:3037] T 67e00dbf892349ffbadfe7527ba92222 P 32d80eed68624cbeb7750d84632e714c [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:35.634510 2586 raft_consensus.cc:2454] T 67e00dbf892349ffbadfe7527ba92222 P 32d80eed68624cbeb7750d84632e714c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25bedb4d9c1f46aeaa1256eb6708adaa in term 1.
I20260929 15:51:35.634536 2309 raft_consensus.cc:2454] T 67e00dbf892349ffbadfe7527ba92222 P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25bedb4d9c1f46aeaa1256eb6708adaa in term 1.
I20260929 15:51:35.634931 2377 leader_election.cc:302] T 67e00dbf892349ffbadfe7527ba92222 P 25bedb4d9c1f46aeaa1256eb6708adaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 25bedb4d9c1f46aeaa1256eb6708adaa, 32d80eed68624cbeb7750d84632e714c, 9d5c2b08b9854b64a4e1491ca9ae1021; no voters:
I20260929 15:51:35.635191 2736 raft_consensus.cc:2781] T 67e00dbf892349ffbadfe7527ba92222 P 25bedb4d9c1f46aeaa1256eb6708adaa [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:51:35.635334 2736 raft_consensus.cc:686] T 67e00dbf892349ffbadfe7527ba92222 P 25bedb4d9c1f46aeaa1256eb6708adaa [term 1 LEADER]: Becoming Leader. State: Replica: 25bedb4d9c1f46aeaa1256eb6708adaa, State: Running, Role: LEADER
I20260929 15:51:35.635632 2736 consensus_queue.cc:237] T 67e00dbf892349ffbadfe7527ba92222 P 25bedb4d9c1f46aeaa1256eb6708adaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } }
I20260929 15:51:35.638370 2146 catalog_manager.cc:5495] T 67e00dbf892349ffbadfe7527ba92222 P 25bedb4d9c1f46aeaa1256eb6708adaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 25bedb4d9c1f46aeaa1256eb6708adaa (127.0.97.2). New cstate: current_term: 1 leader_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25bedb4d9c1f46aeaa1256eb6708adaa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } health_report { overall_health: UNKNOWN } } }
I20260929 15:51:35.671130 2774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:35.665975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54684 (local address 127.0.97.2:33143)
0929 15:51:35.666574 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:35.666578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:35.666595 (+ 17us) server_negotiation.cc:407] Connection header received
0929 15:51:35.666639 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:35.666642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:35.666694 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:35.666798 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:51:35.667206 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:35.667951 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:35.669229 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:35.669796 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:35.669926 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:35.670337 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:51:35.670450 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:51:35.670543 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:35.670704 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 15:51:35.670787 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":326,"threads_started":1}
I20260929 15:51:35.680114 2586 raft_consensus.cc:1261] T 67e00dbf892349ffbadfe7527ba92222 P 32d80eed68624cbeb7750d84632e714c [term 1 FOLLOWER]: Refusing update from remote peer 25bedb4d9c1f46aeaa1256eb6708adaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:51:35.680212 2309 raft_consensus.cc:1261] T 67e00dbf892349ffbadfe7527ba92222 P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 1 FOLLOWER]: Refusing update from remote peer 25bedb4d9c1f46aeaa1256eb6708adaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:51:35.680673 2733 consensus_queue.cc:1035] T 67e00dbf892349ffbadfe7527ba92222 P 25bedb4d9c1f46aeaa1256eb6708adaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "32d80eed68624cbeb7750d84632e714c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:51:35.681135 2733 consensus_queue.cc:1035] T 67e00dbf892349ffbadfe7527ba92222 P 25bedb4d9c1f46aeaa1256eb6708adaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d5c2b08b9854b64a4e1491ca9ae1021" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 15:51:35.703234 2724 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 15:51:37.290642 2146 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:51:37.290858 2146 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:51:37.294441 2146 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8087fc375014469989c805e973593c1b: Sending DeleteTablet for 3 replicas of tablet 67e00dbf892349ffbadfe7527ba92222
I20260929 15:51:37.295146 2283 tablet_service.cc:1514] Processing DeleteTablet for tablet 67e00dbf892349ffbadfe7527ba92222 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:51:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50954
I20260929 15:51:37.295141 2423 tablet_service.cc:1514] Processing DeleteTablet for tablet 67e00dbf892349ffbadfe7527ba92222 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:51:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54660
I20260929 15:51:37.295394 2562 tablet_service.cc:1514] Processing DeleteTablet for tablet 67e00dbf892349ffbadfe7527ba92222 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:51:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59922
I20260929 15:51:37.295863 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 2218
I20260929 15:51:37.311045 2801 tablet_replica.cc:330] T 67e00dbf892349ffbadfe7527ba92222 P 9d5c2b08b9854b64a4e1491ca9ae1021: stopping tablet replica
I20260929 15:51:37.311721 2802 tablet_replica.cc:330] T 67e00dbf892349ffbadfe7527ba92222 P 32d80eed68624cbeb7750d84632e714c: stopping tablet replica
I20260929 15:51:37.312155 2802 raft_consensus.cc:2229] T 67e00dbf892349ffbadfe7527ba92222 P 32d80eed68624cbeb7750d84632e714c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:51:37.312561 2801 raft_consensus.cc:2229] T 67e00dbf892349ffbadfe7527ba92222 P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:51:37.312922 2801 raft_consensus.cc:2258] T 67e00dbf892349ffbadfe7527ba92222 P 9d5c2b08b9854b64a4e1491ca9ae1021 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:51:37.313376 2802 raft_consensus.cc:2258] T 67e00dbf892349ffbadfe7527ba92222 P 32d80eed68624cbeb7750d84632e714c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:51:37.314767 2800 tablet_replica.cc:330] T 67e00dbf892349ffbadfe7527ba92222 P 25bedb4d9c1f46aeaa1256eb6708adaa: stopping tablet replica
I20260929 15:51:37.314963 2800 raft_consensus.cc:2229] T 67e00dbf892349ffbadfe7527ba92222 P 25bedb4d9c1f46aeaa1256eb6708adaa [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:51:37.315305 2800 raft_consensus.cc:2258] T 67e00dbf892349ffbadfe7527ba92222 P 25bedb4d9c1f46aeaa1256eb6708adaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:51:37.316205 2801 ts_tablet_manager.cc:1905] T 67e00dbf892349ffbadfe7527ba92222 P 9d5c2b08b9854b64a4e1491ca9ae1021: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:51:37.316217 2800 ts_tablet_manager.cc:1905] T 67e00dbf892349ffbadfe7527ba92222 P 25bedb4d9c1f46aeaa1256eb6708adaa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:51:37.316532 2802 ts_tablet_manager.cc:1905] T 67e00dbf892349ffbadfe7527ba92222 P 32d80eed68624cbeb7750d84632e714c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:51:37.322959 2801 ts_tablet_manager.cc:1918] T 67e00dbf892349ffbadfe7527ba92222 P 9d5c2b08b9854b64a4e1491ca9ae1021: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 15:51:37.322959 2800 ts_tablet_manager.cc:1918] T 67e00dbf892349ffbadfe7527ba92222 P 25bedb4d9c1f46aeaa1256eb6708adaa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 15:51:37.323082 2800 log.cc:1196] T 67e00dbf892349ffbadfe7527ba92222 P 25bedb4d9c1f46aeaa1256eb6708adaa: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-1/wal/wals/67e00dbf892349ffbadfe7527ba92222
I20260929 15:51:37.323211 2801 log.cc:1196] T 67e00dbf892349ffbadfe7527ba92222 P 9d5c2b08b9854b64a4e1491ca9ae1021: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-0/wal/wals/67e00dbf892349ffbadfe7527ba92222
I20260929 15:51:37.323501 2800 ts_tablet_manager.cc:1939] T 67e00dbf892349ffbadfe7527ba92222 P 25bedb4d9c1f46aeaa1256eb6708adaa: Deleting consensus metadata
I20260929 15:51:37.323730 2801 ts_tablet_manager.cc:1939] T 67e00dbf892349ffbadfe7527ba92222 P 9d5c2b08b9854b64a4e1491ca9ae1021: Deleting consensus metadata
Sep 29 15:51:37 dist-test-slave-r0ck krb5kdc[2102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697095, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 15:51:37.324577 2132 catalog_manager.cc:4841] TS 25bedb4d9c1f46aeaa1256eb6708adaa (127.0.97.2:33143): tablet 67e00dbf892349ffbadfe7527ba92222 (table test-table [id=bb3d829b9c1444058e34285fc25849e2]) successfully deleted
I20260929 15:51:37.324741 2802 ts_tablet_manager.cc:1918] T 67e00dbf892349ffbadfe7527ba92222 P 32d80eed68624cbeb7750d84632e714c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 15:51:37.324805 2131 catalog_manager.cc:4841] TS 9d5c2b08b9854b64a4e1491ca9ae1021 (127.0.97.1:37035): tablet 67e00dbf892349ffbadfe7527ba92222 (table test-table [id=bb3d829b9c1444058e34285fc25849e2]) successfully deleted
I20260929 15:51:37.324887 2802 log.cc:1196] T 67e00dbf892349ffbadfe7527ba92222 P 32d80eed68624cbeb7750d84632e714c: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697075657932-388-0/minicluster-data/ts-2/wal/wals/67e00dbf892349ffbadfe7527ba92222
I20260929 15:51:37.325395 2802 ts_tablet_manager.cc:1939] T 67e00dbf892349ffbadfe7527ba92222 P 32d80eed68624cbeb7750d84632e714c: Deleting consensus metadata
I20260929 15:51:37.326493 2134 catalog_manager.cc:4841] TS 32d80eed68624cbeb7750d84632e714c (127.0.97.3:43617): tablet 67e00dbf892349ffbadfe7527ba92222 (table test-table [id=bb3d829b9c1444058e34285fc25849e2]) successfully deleted
I20260929 15:51:37.330628 2804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:37.297579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50988 (local address 127.0.97.1:37035)
0929 15:51:37.312558 (+ 14979us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:37.312562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:37.312582 (+ 20us) server_negotiation.cc:407] Connection header received
0929 15:51:37.312627 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:37.312631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:37.312720 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:37.312843 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:37.314516 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:37.315781 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:37.317962 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:37.318717 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:37.318847 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:37.325118 (+ 6271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:37.325130 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:37.325146 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:37.325174 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:37.328102 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:37.328739 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:37.328746 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:37.328748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:37.328900 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:37.329256 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:37.329260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:37.329262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:37.329694 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:37.329962 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:37.330124 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 15:51:37.330277 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14861,"thread_start_us":14740,"threads_started":1}
W20260929 15:51:37.331235 2349 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50988
W20260929 15:51:37.331781 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 2218: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:51:37.332023 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 2218
I20260929 15:51:37.359300 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 2358
Sep 29 15:51:37 dist-test-slave-r0ck krb5kdc[2102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697095, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 15:51:37.374522 2805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:37.360187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48692 (local address 127.0.97.2:33143)
0929 15:51:37.361052 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:37.361057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:37.361074 (+ 17us) server_negotiation.cc:407] Connection header received
0929 15:51:37.361132 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:37.361135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:37.361206 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:37.361299 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:37.362564 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:37.363316 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:37.364480 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:37.364906 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:37.365057 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:37.369332 (+ 4275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:37.369391 (+ 59us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:37.369417 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:37.369450 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:37.372114 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:37.372734 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:37.372741 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:37.372743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:37.372886 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:37.373175 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:37.373178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:37.373180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:37.373606 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:37.373800 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:37.374097 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 15:51:37.374256 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":407,"threads_started":1}
W20260929 15:51:37.375187 2488 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48692
W20260929 15:51:37.375813 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 2358: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:51:37.375926 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 2358
I20260929 15:51:37.397755 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 2497
Sep 29 15:51:37 dist-test-slave-r0ck krb5kdc[2102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697095, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 15:51:37.412997 2806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:37.398610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58240 (local address 127.0.97.3:43617)
0929 15:51:37.399107 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:37.399112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:37.399128 (+ 16us) server_negotiation.cc:407] Connection header received
0929 15:51:37.399179 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:37.399183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:37.399239 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:37.399385 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:37.400549 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:37.401327 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:37.402558 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:37.402940 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:37.403036 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:37.407215 (+ 4179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:37.407229 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:37.407245 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:37.407280 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:37.410345 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:37.411021 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:37.411028 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:37.411031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:37.411164 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:37.411529 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:37.411533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:37.411535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:37.411970 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:37.412208 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:37.412557 (+ 349us) server_negotiation.cc:299] Negotiation successful
0929 15:51:37.412726 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":286,"threads_started":1}
W20260929 15:51:37.414047 2627 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58240
W20260929 15:51:37.414880 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 2497: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:51:37.415076 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 2497
I20260929 15:51:37.436576 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 2115
I20260929 15:51:37.437861 2187 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:51:37.552067 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 2115
2026-09-29T15:51:37Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8061 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:51:37 dist-test-slave-r0ck krb5kdc[2810](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:51:37 dist-test-slave-r0ck krb5kdc[2810](info): set up 2 sockets
Sep 29 15:51:37 dist-test-slave-r0ck krb5kdc[2810](info): commencing operation
krb5kdc: starting...
W20260929 15:51:38.645720 388 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:51:38 dist-test-slave-r0ck krb5kdc[2810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:51:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:51:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:51:38.825423 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44159
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44159
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:51:39.160826 2823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:51:39.161123 2823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:51:39.161290 2823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:51:39.168524 2823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:51:39.168629 2823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:51:39.168715 2823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:51:39.168838 2823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:51:39.168916 2823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:51:39.178849 2823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:51:39.180222 2823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:51:39.181581 2823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:51:39.193908 2828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:39.193882 2829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:39.193858 2831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:51:39.195082 2823 server_base.cc:1034] running on GCE node
I20260929 15:51:39.195812 2823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:51:39.197235 2823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:51:39.198429 2823 hybrid_clock.cc:648] HybridClock initialized: now 1790697099198368 us; error 55 us; skew 500 ppm
I20260929 15:51:39.198766 2823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:51:39 dist-test-slave-r0ck krb5kdc[2810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:51:39.201730 2823 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:51:39.203250 2823 webserver.cc:460] Webserver started at http://127.0.97.62:32995/ using document root <none> and password file <none>
I20260929 15:51:39.203912 2823 fs_manager.cc:359] Metadata directory not provided
I20260929 15:51:39.204058 2823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:51:39.204361 2823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:51:39.207221 2823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "f11c62633dbe4f7e8b8f5d2798622bf4"
format_stamp: "Formatted at 2026-09-29 15:51:39 on dist-test-slave-r0ck"
server_key: "6bfa8e3de23115e736606e3c698e2334"
server_key_iv: "6645cd981378e88125f8f79974bf7ae7"
server_key_version: "encryptionkey@0"
I20260929 15:51:39.207965 2823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "f11c62633dbe4f7e8b8f5d2798622bf4"
format_stamp: "Formatted at 2026-09-29 15:51:39 on dist-test-slave-r0ck"
server_key: "6bfa8e3de23115e736606e3c698e2334"
server_key_iv: "6645cd981378e88125f8f79974bf7ae7"
server_key_version: "encryptionkey@0"
I20260929 15:51:39.212819 2823 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260929 15:51:39.216246 2838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:39.217644 2823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 15:51:39.217824 2823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "f11c62633dbe4f7e8b8f5d2798622bf4"
format_stamp: "Formatted at 2026-09-29 15:51:39 on dist-test-slave-r0ck"
server_key: "6bfa8e3de23115e736606e3c698e2334"
server_key_iv: "6645cd981378e88125f8f79974bf7ae7"
server_key_version: "encryptionkey@0"
I20260929 15:51:39.218025 2823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:51:39.240168 2823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:51:39.244078 2823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:51:39.244359 2823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:51:39.262310 2823 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44159
I20260929 15:51:39.262351 2889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44159 every 8 connection(s)
I20260929 15:51:39.263584 2823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:51:39.267104 2890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:39.272038 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 2823
I20260929 15:51:39.272257 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:51:39.272984 388 external_mini_cluster.cc:1428] Setting key 41d0a417c81b3fcd1c4a441643a4091e
I20260929 15:51:39.278117 2890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f11c62633dbe4f7e8b8f5d2798622bf4: Bootstrap starting.
I20260929 15:51:39.281596 2890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f11c62633dbe4f7e8b8f5d2798622bf4: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:39.282738 2890 log.cc:824] T 00000000000000000000000000000000 P f11c62633dbe4f7e8b8f5d2798622bf4: Log is configured to *not* fsync() on all Append() calls
I20260929 15:51:39.285996 2890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f11c62633dbe4f7e8b8f5d2798622bf4: No bootstrap required, opened a new log
Sep 29 15:51:39 dist-test-slave-r0ck krb5kdc[2810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:51:39.293566 2890 raft_consensus.cc:348] T 00000000000000000000000000000000 P f11c62633dbe4f7e8b8f5d2798622bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11c62633dbe4f7e8b8f5d2798622bf4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44159 } }
I20260929 15:51:39.293928 2890 raft_consensus.cc:374] T 00000000000000000000000000000000 P f11c62633dbe4f7e8b8f5d2798622bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:39.294054 2890 raft_consensus.cc:729] T 00000000000000000000000000000000 P f11c62633dbe4f7e8b8f5d2798622bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f11c62633dbe4f7e8b8f5d2798622bf4, State: Initialized, Role: FOLLOWER
I20260929 15:51:39.294772 2890 consensus_queue.cc:260] T 00000000000000000000000000000000 P f11c62633dbe4f7e8b8f5d2798622bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11c62633dbe4f7e8b8f5d2798622bf4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44159 } }
I20260929 15:51:39.295047 2890 raft_consensus.cc:388] T 00000000000000000000000000000000 P f11c62633dbe4f7e8b8f5d2798622bf4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:51:39.295193 2890 raft_consensus.cc:482] T 00000000000000000000000000000000 P f11c62633dbe4f7e8b8f5d2798622bf4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:51:39.295377 2890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f11c62633dbe4f7e8b8f5d2798622bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:39.295647 2893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:39.274958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36242 (local address 127.0.97.62:44159)
0929 15:51:39.275751 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:39.275763 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:39.275804 (+ 41us) server_negotiation.cc:407] Connection header received
0929 15:51:39.277473 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:39.277493 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:39.277848 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:39.278141 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:39.279854 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:39.281112 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:39.282959 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:39.284019 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:39.284150 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:39.288539 (+ 4389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:39.288547 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:39.288562 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:39.288596 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:39.291615 (+ 3019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:39.292271 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:39.292278 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:39.292283 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:39.292456 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:39.292835 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:39.292840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:39.292844 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:39.293322 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:39.293664 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:39.294089 (+ 425us) server_negotiation.cc:299] Negotiation successful
0929 15:51:39.294485 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":360,"threads_started":1}
I20260929 15:51:39.297780 2890 raft_consensus.cc:504] T 00000000000000000000000000000000 P f11c62633dbe4f7e8b8f5d2798622bf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11c62633dbe4f7e8b8f5d2798622bf4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44159 } }
I20260929 15:51:39.298316 2890 leader_election.cc:302] T 00000000000000000000000000000000 P f11c62633dbe4f7e8b8f5d2798622bf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f11c62633dbe4f7e8b8f5d2798622bf4; no voters:
I20260929 15:51:39.298864 2890 leader_election.cc:288] T 00000000000000000000000000000000 P f11c62633dbe4f7e8b8f5d2798622bf4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:51:39.299009 2895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f11c62633dbe4f7e8b8f5d2798622bf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:51:39.299665 2895 raft_consensus.cc:686] T 00000000000000000000000000000000 P f11c62633dbe4f7e8b8f5d2798622bf4 [term 1 LEADER]: Becoming Leader. State: Replica: f11c62633dbe4f7e8b8f5d2798622bf4, State: Running, Role: LEADER
I20260929 15:51:39.300168 2895 consensus_queue.cc:237] T 00000000000000000000000000000000 P f11c62633dbe4f7e8b8f5d2798622bf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11c62633dbe4f7e8b8f5d2798622bf4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44159 } }
I20260929 15:51:39.300863 2890 sys_catalog.cc:564] T 00000000000000000000000000000000 P f11c62633dbe4f7e8b8f5d2798622bf4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:51:39.302999 2897 sys_catalog.cc:455] T 00000000000000000000000000000000 P f11c62633dbe4f7e8b8f5d2798622bf4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f11c62633dbe4f7e8b8f5d2798622bf4. Latest consensus state: current_term: 1 leader_uuid: "f11c62633dbe4f7e8b8f5d2798622bf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11c62633dbe4f7e8b8f5d2798622bf4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44159 } } }
I20260929 15:51:39.303443 2897 sys_catalog.cc:458] T 00000000000000000000000000000000 P f11c62633dbe4f7e8b8f5d2798622bf4 [sys.catalog]: This master's current role is: LEADER
I20260929 15:51:39.302994 2896 sys_catalog.cc:455] T 00000000000000000000000000000000 P f11c62633dbe4f7e8b8f5d2798622bf4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f11c62633dbe4f7e8b8f5d2798622bf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11c62633dbe4f7e8b8f5d2798622bf4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44159 } } }
I20260929 15:51:39.303841 2896 sys_catalog.cc:458] T 00000000000000000000000000000000 P f11c62633dbe4f7e8b8f5d2798622bf4 [sys.catalog]: This master's current role is: LEADER
I20260929 15:51:39.305197 2900 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:51:39.310256 2900 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:51:39.318857 2900 catalog_manager.cc:1347] Generated new cluster ID: af691b43cff5429db5233d555d034c76
I20260929 15:51:39.318961 2900 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:51:39.341410 2900 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:51:39.343211 2900 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:51:39.352623 2900 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f11c62633dbe4f7e8b8f5d2798622bf4: Generated new TSK 0
I20260929 15:51:39.353513 2900 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 15:51:39.414965 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44159
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:51:39.743247 2916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:51:39.743501 2916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:51:39.743656 2916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:51:39.750813 2916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:51:39.750908 2916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:51:39.751080 2916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:51:39.760272 2916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44159
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:51:39.761507 2916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:51:39.762766 2916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:51:39.775264 2921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:39.775242 2924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:39.775305 2922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:51:39.776167 2916 server_base.cc:1034] running on GCE node
I20260929 15:51:39.776743 2916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:51:39.777675 2916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:51:39.778903 2916 hybrid_clock.cc:648] HybridClock initialized: now 1790697099778863 us; error 49 us; skew 500 ppm
I20260929 15:51:39.779227 2916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:51:39 dist-test-slave-r0ck krb5kdc[2810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:51:39.782205 2916 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:51:39.783560 2916 webserver.cc:460] Webserver started at http://127.0.97.1:42461/ using document root <none> and password file <none>
I20260929 15:51:39.784147 2916 fs_manager.cc:359] Metadata directory not provided
I20260929 15:51:39.784264 2916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:51:39.784576 2916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:51:39.787289 2916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-0/data/instance:
uuid: "d341e742f5e943a1bf3cdf0575c7b110"
format_stamp: "Formatted at 2026-09-29 15:51:39 on dist-test-slave-r0ck"
server_key: "fab3814d87b40dfd479ce7d1b5f91a0f"
server_key_iv: "a9627512c7851ad43bcc9ab7ffe3790c"
server_key_version: "encryptionkey@0"
I20260929 15:51:39.788029 2916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d341e742f5e943a1bf3cdf0575c7b110"
format_stamp: "Formatted at 2026-09-29 15:51:39 on dist-test-slave-r0ck"
server_key: "fab3814d87b40dfd479ce7d1b5f91a0f"
server_key_iv: "a9627512c7851ad43bcc9ab7ffe3790c"
server_key_version: "encryptionkey@0"
I20260929 15:51:39.792888 2916 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260929 15:51:39.796262 2931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:39.797627 2916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:51:39.797784 2916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-0/wal
uuid: "d341e742f5e943a1bf3cdf0575c7b110"
format_stamp: "Formatted at 2026-09-29 15:51:39 on dist-test-slave-r0ck"
server_key: "fab3814d87b40dfd479ce7d1b5f91a0f"
server_key_iv: "a9627512c7851ad43bcc9ab7ffe3790c"
server_key_version: "encryptionkey@0"
I20260929 15:51:39.797972 2916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:51:39.813416 2916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:51:39.817467 2916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:51:39.817761 2916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:51:39.818676 2916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:51:39.820084 2916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:51:39.820182 2916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:39.820333 2916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:51:39.820407 2916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:39.853174 2916 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36101
I20260929 15:51:39.853215 3043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36101 every 8 connection(s)
I20260929 15:51:39.854444 2916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:51:39.863873 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 2916
I20260929 15:51:39.864099 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:51:39.864715 388 external_mini_cluster.cc:1428] Setting key d099ab67ad9e27d76db6cdfb9fd33025
Sep 29 15:51:39 dist-test-slave-r0ck krb5kdc[2810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:51:39.875043 3046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:39.856963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35015 (local address 127.0.97.62:44159)
0929 15:51:39.857441 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:39.857445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:39.858247 (+ 802us) server_negotiation.cc:407] Connection header received
0929 15:51:39.860030 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:39.860035 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:39.860121 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:39.860234 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:39.861926 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:39.862657 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:39.864374 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:39.864774 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:39.864919 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:39.868892 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:39.868897 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:39.868900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:39.868934 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:39.871715 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:39.872561 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:39.872566 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:39.872568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:39.872687 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:39.873142 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:39.873149 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:39.873158 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:39.873506 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:39.873675 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:39.874417 (+ 742us) server_negotiation.cc:299] Negotiation successful
0929 15:51:39.874659 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":236,"threads_started":1}
I20260929 15:51:39.876051 3047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:39.857425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44159 (local address 127.0.97.1:35015)
0929 15:51:39.858101 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:39.858137 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:39.859755 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:39.860458 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:39.860466 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:39.860957 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:51:39.861591 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:39.861611 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:39.862818 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:39.862831 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:39.864229 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:39.864241 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:39.865069 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:39.865077 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:39.865431 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:39.866220 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:51:39.866243 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:51:39.868622 (+ 2379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:51:39.871917 (+ 3295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:39.871931 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:39.871947 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:39.872404 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:39.872805 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:39.872813 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:39.872817 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:39.873019 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:39.873639 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:51:39.873650 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:51:39.874034 (+ 384us) client_negotiation.cc:769] Sending connection context
0929 15:51:39.874314 (+ 280us) client_negotiation.cc:240] Negotiation successful
0929 15:51:39.874884 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":256,"threads_started":1}
I20260929 15:51:39.877815 3044 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44159
I20260929 15:51:39.878160 3044 heartbeater.cc:464] Registering TS with master...
I20260929 15:51:39.879076 3044 heartbeater.cc:511] Master 127.0.97.62:44159 requested a full tablet report, sending...
I20260929 15:51:39.881762 2855 ts_manager.cc:194] Registered new tserver with Master: d341e742f5e943a1bf3cdf0575c7b110 (127.0.97.1:36101)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 15:51:39.883980 2855 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:35015
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 15:51:39.909107 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44159
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:51:40.225143 3050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:51:40.225471 3050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:51:40.225620 3050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:51:40.232748 3050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:51:40.232862 3050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:51:40.233031 3050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:51:40.242905 3050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44159
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:51:40.244117 3050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:51:40.245436 3050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:51:40.258582 3056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:40.258580 3058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:40.258847 3055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:51:40.259517 3050 server_base.cc:1034] running on GCE node
I20260929 15:51:40.260159 3050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:51:40.261088 3050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:51:40.262346 3050 hybrid_clock.cc:648] HybridClock initialized: now 1790697100262259 us; error 60 us; skew 500 ppm
I20260929 15:51:40.262877 3050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:51:40 dist-test-slave-r0ck krb5kdc[2810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:51:40.266812 3050 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:51:40.278558 3050 webserver.cc:460] Webserver started at http://127.0.97.2:45431/ using document root <none> and password file <none>
I20260929 15:51:40.279289 3050 fs_manager.cc:359] Metadata directory not provided
I20260929 15:51:40.279440 3050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:51:40.279752 3050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:51:40.282718 3050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-1/data/instance:
uuid: "ff41dbb733354f9aa450b015a1e2084b"
format_stamp: "Formatted at 2026-09-29 15:51:40 on dist-test-slave-r0ck"
server_key: "cb9b968fc226dc4c43037ce77ca576df"
server_key_iv: "6245c13648f3acdb246cc6a41db9fa58"
server_key_version: "encryptionkey@0"
I20260929 15:51:40.283496 3050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ff41dbb733354f9aa450b015a1e2084b"
format_stamp: "Formatted at 2026-09-29 15:51:40 on dist-test-slave-r0ck"
server_key: "cb9b968fc226dc4c43037ce77ca576df"
server_key_iv: "6245c13648f3acdb246cc6a41db9fa58"
server_key_version: "encryptionkey@0"
I20260929 15:51:40.288609 3050 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260929 15:51:40.291909 3065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:40.293124 3050 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 15:51:40.293295 3050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-1/wal
uuid: "ff41dbb733354f9aa450b015a1e2084b"
format_stamp: "Formatted at 2026-09-29 15:51:40 on dist-test-slave-r0ck"
server_key: "cb9b968fc226dc4c43037ce77ca576df"
server_key_iv: "6245c13648f3acdb246cc6a41db9fa58"
server_key_version: "encryptionkey@0"
I20260929 15:51:40.293540 3050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:51:40.309142 3050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:51:40.313436 3050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:51:40.313710 3050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:51:40.314615 3050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:51:40.315873 3050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:51:40.315971 3050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:40.316114 3050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:51:40.316186 3050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:40.350683 3050 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35215
I20260929 15:51:40.350759 3177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35215 every 8 connection(s)
I20260929 15:51:40.352377 3050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:51:40.357229 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 3050
I20260929 15:51:40.357452 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:51:40.357868 388 external_mini_cluster.cc:1428] Setting key e1b1bca5e80cf666692956cd568f5cf5
Sep 29 15:51:40 dist-test-slave-r0ck krb5kdc[2810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:51:40.374989 3046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:40.355129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:32979 (local address 127.0.97.62:44159)
0929 15:51:40.355292 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:40.355296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:40.356671 (+ 1375us) server_negotiation.cc:407] Connection header received
0929 15:51:40.358618 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:40.358623 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:40.358706 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:40.358813 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:40.360326 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:40.361507 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:40.363079 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:40.363845 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:40.363988 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:40.367989 (+ 4001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:40.367998 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:40.368006 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:40.368051 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:40.371729 (+ 3678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:40.372450 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:40.372458 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:40.372464 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:40.372629 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:40.373059 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:40.373064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:40.373068 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:40.373466 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:40.373632 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:40.374318 (+ 686us) server_negotiation.cc:299] Negotiation successful
0929 15:51:40.374567 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 15:51:40.376185 3180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:40.355629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44159 (local address 127.0.97.2:32979)
0929 15:51:40.356498 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:40.356535 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:40.358356 (+ 1821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:40.359004 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:40.359014 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:40.359599 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:51:40.360148 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:40.360166 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:40.361685 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:40.361690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:40.362922 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:40.362934 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:40.364077 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:40.364149 (+ 72us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:40.364383 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:40.365083 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:51:40.365116 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:51:40.367761 (+ 2645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:51:40.371871 (+ 4110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:40.371877 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:40.371890 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:40.372325 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:40.372754 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:40.372758 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:40.372760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:40.372947 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:40.373613 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:51:40.373619 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:51:40.373930 (+ 311us) client_negotiation.cc:769] Sending connection context
0929 15:51:40.374182 (+ 252us) client_negotiation.cc:240] Negotiation successful
0929 15:51:40.374707 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":437,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 15:51:40.378087 3178 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44159
I20260929 15:51:40.378360 3178 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 15:51:40.379273 3178 heartbeater.cc:511] Master 127.0.97.62:44159 requested a full tablet report, sending...
I20260929 15:51:40.380867 2855 ts_manager.cc:194] Registered new tserver with Master: ff41dbb733354f9aa450b015a1e2084b (127.0.97.2:35215)
I20260929 15:51:40.381757 2855 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:32979
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 15:51:40.401624 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44159
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:51:40.713858 3183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:51:40.714221 3183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:51:40.714406 3183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:51:40.721637 3183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:51:40.721783 3183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:51:40.721961 3183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:51:40.731602 3183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44159
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:51:40.732834 3183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:51:40.734153 3183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:51:40.746788 3189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:40.746759 3188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:40.746824 3191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:51:40.747716 3183 server_base.cc:1034] running on GCE node
I20260929 15:51:40.748227 3183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:51:40.749145 3183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:51:40.750357 3183 hybrid_clock.cc:648] HybridClock initialized: now 1790697100750308 us; error 45 us; skew 500 ppm
I20260929 15:51:40.750620 3183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:51:40 dist-test-slave-r0ck krb5kdc[2810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:51:40.753641 3183 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:51:40.755041 3183 webserver.cc:460] Webserver started at http://127.0.97.3:46759/ using document root <none> and password file <none>
I20260929 15:51:40.755681 3183 fs_manager.cc:359] Metadata directory not provided
I20260929 15:51:40.755828 3183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:51:40.756127 3183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:51:40.759030 3183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-2/data/instance:
uuid: "a4450bdc5efb4214ba47baeaaaa0b84d"
format_stamp: "Formatted at 2026-09-29 15:51:40 on dist-test-slave-r0ck"
server_key: "c1e8d0a6a4c5c27e9384677b1fac9e5a"
server_key_iv: "89c5242d81070590c576b8ca94a3b4d0"
server_key_version: "encryptionkey@0"
I20260929 15:51:40.759788 3183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a4450bdc5efb4214ba47baeaaaa0b84d"
format_stamp: "Formatted at 2026-09-29 15:51:40 on dist-test-slave-r0ck"
server_key: "c1e8d0a6a4c5c27e9384677b1fac9e5a"
server_key_iv: "89c5242d81070590c576b8ca94a3b4d0"
server_key_version: "encryptionkey@0"
I20260929 15:51:40.764663 3183 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 15:51:40.768115 3198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:40.769419 3183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 15:51:40.769575 3183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-2/wal
uuid: "a4450bdc5efb4214ba47baeaaaa0b84d"
format_stamp: "Formatted at 2026-09-29 15:51:40 on dist-test-slave-r0ck"
server_key: "c1e8d0a6a4c5c27e9384677b1fac9e5a"
server_key_iv: "89c5242d81070590c576b8ca94a3b4d0"
server_key_version: "encryptionkey@0"
I20260929 15:51:40.769764 3183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:51:40.795027 3183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:51:40.799152 3183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:51:40.799430 3183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:51:40.800308 3183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:51:40.801878 3183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:51:40.801975 3183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:40.802120 3183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:51:40.802191 3183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:40.835289 3183 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42709
I20260929 15:51:40.835376 3310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42709 every 8 connection(s)
I20260929 15:51:40.836570 3183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:51:40.838362 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 3183
I20260929 15:51:40.838555 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:51:40.838986 388 external_mini_cluster.cc:1428] Setting key ebc2fa8c8eefe854b9ae4d513586b470
Sep 29 15:51:40 dist-test-slave-r0ck krb5kdc[2810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:51:40.857184 3046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:40.839371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58781 (local address 127.0.97.62:44159)
0929 15:51:40.839584 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:40.839590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:40.840892 (+ 1302us) server_negotiation.cc:407] Connection header received
0929 15:51:40.843061 (+ 2169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:40.843066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:40.843149 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:40.843266 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:40.844671 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:40.845514 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:40.847031 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:40.847430 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:40.847564 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:40.850863 (+ 3299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:40.850869 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:40.850874 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:40.850915 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:40.854201 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:40.854935 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:40.854941 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:40.854944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:40.855086 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:40.855485 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:40.855490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:40.855495 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:40.855836 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:40.855980 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:40.856634 (+ 654us) server_negotiation.cc:299] Negotiation successful
0929 15:51:40.856845 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 15:51:40.858060 3313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:40.839787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44159 (local address 127.0.97.3:58781)
0929 15:51:40.840704 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:40.840768 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:40.842798 (+ 2030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:40.843491 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:40.843498 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:40.843983 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:51:40.844506 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:40.844521 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:40.845660 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:40.845672 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:40.846867 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:40.846879 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:40.847644 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:40.847653 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:40.847885 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:40.848573 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:51:40.848594 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:51:40.850665 (+ 2071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:51:40.854370 (+ 3705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:40.854378 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:40.854392 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:40.854814 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:40.855193 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:40.855196 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:40.855199 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:40.855376 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:40.855962 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:51:40.855967 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:51:40.856257 (+ 290us) client_negotiation.cc:769] Sending connection context
0929 15:51:40.856510 (+ 253us) client_negotiation.cc:240] Negotiation successful
0929 15:51:40.856869 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":381,"threads_started":1}
I20260929 15:51:40.859438 3311 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44159
I20260929 15:51:40.859732 3311 heartbeater.cc:464] Registering TS with master...
I20260929 15:51:40.860409 3311 heartbeater.cc:511] Master 127.0.97.62:44159 requested a full tablet report, sending...
I20260929 15:51:40.861871 2855 ts_manager.cc:194] Registered new tserver with Master: a4450bdc5efb4214ba47baeaaaa0b84d (127.0.97.3:42709)
I20260929 15:51:40.862677 2855 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:58781
I20260929 15:51:40.867148 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:51:40.887531 3044 heartbeater.cc:503] Master 127.0.97.62:44159 was elected leader, sending a full tablet report...
W20260929 15:51:40.897980 3046 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:51:40.895424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36258 (local address 127.0.97.62:44159)
0929 15:51:40.895613 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:40.895617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:40.896095 (+ 478us) server_negotiation.cc:407] Connection header received
0929 15:51:40.896246 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:40.896249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:40.896312 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:40.896413 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:40.897736 (+ 1323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36258: BlockingRecv error: recv got EOF from 127.0.0.1:36258 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 15:51:40.900879 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 2916
W20260929 15:51:40.903986 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 2916: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:36101: server requires authentication, but client does not have Kerberos credentials available
I20260929 15:51:40.904170 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 2916
I20260929 15:51:40.919437 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 3050
W20260929 15:51:40.922084 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 3050: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:35215: server requires authentication, but client does not have Kerberos credentials available
I20260929 15:51:40.922241 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 3050
I20260929 15:51:40.937320 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 3183
W20260929 15:51:40.940409 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 3183: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:42709: server requires authentication, but client does not have Kerberos credentials available
I20260929 15:51:40.940563 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 3183
I20260929 15:51:40.956007 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 2823
I20260929 15:51:40.957297 2885 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:51:41.061081 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 2823
2026-09-29T15:51:41Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3506 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:51:41 dist-test-slave-r0ck krb5kdc[3328](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:51:41 dist-test-slave-r0ck krb5kdc[3328](info): set up 2 sockets
Sep 29 15:51:41 dist-test-slave-r0ck krb5kdc[3328](info): commencing operation
krb5kdc: starting...
W20260929 15:51:42.145715 388 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:51:42 dist-test-slave-r0ck krb5kdc[3328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:51:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:51:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:51:42.330444 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33225
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:51:42.641628 3341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:51:42.641894 3341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:51:42.641981 3341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:51:42.648957 3341 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:51:42.649060 3341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:51:42.649104 3341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:51:42.649152 3341 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:51:42.649199 3341 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:51:42.658733 3341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:51:42.659902 3341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:51:42.661118 3341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:51:42.672710 3346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:42.672669 3349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:42.672752 3347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:51:42.673803 3341 server_base.cc:1034] running on GCE node
I20260929 15:51:42.674576 3341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:51:42.675967 3341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:51:42.677194 3341 hybrid_clock.cc:648] HybridClock initialized: now 1790697102677148 us; error 43 us; skew 500 ppm
I20260929 15:51:42.677593 3341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:51:42 dist-test-slave-r0ck krb5kdc[3328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:51:42.680606 3341 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:51:42.682163 3341 webserver.cc:460] Webserver started at http://127.0.97.62:36967/ using document root <none> and password file <none>
I20260929 15:51:42.682835 3341 fs_manager.cc:359] Metadata directory not provided
I20260929 15:51:42.682981 3341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:51:42.683274 3341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:51:42.686200 3341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "d0d8b46c768746fbabaaeaf8f739db3c"
format_stamp: "Formatted at 2026-09-29 15:51:42 on dist-test-slave-r0ck"
server_key: "7787401f363507bc9c1413cd3757d54a"
server_key_iv: "ebe28653ed47e0b6c67c9339c4006f45"
server_key_version: "encryptionkey@0"
I20260929 15:51:42.686964 3341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "d0d8b46c768746fbabaaeaf8f739db3c"
format_stamp: "Formatted at 2026-09-29 15:51:42 on dist-test-slave-r0ck"
server_key: "7787401f363507bc9c1413cd3757d54a"
server_key_iv: "ebe28653ed47e0b6c67c9339c4006f45"
server_key_version: "encryptionkey@0"
I20260929 15:51:42.691803 3341 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 15:51:42.695045 3356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:42.696408 3341 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 15:51:42.696573 3341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "d0d8b46c768746fbabaaeaf8f739db3c"
format_stamp: "Formatted at 2026-09-29 15:51:42 on dist-test-slave-r0ck"
server_key: "7787401f363507bc9c1413cd3757d54a"
server_key_iv: "ebe28653ed47e0b6c67c9339c4006f45"
server_key_version: "encryptionkey@0"
I20260929 15:51:42.696756 3341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:51:42.724237 3341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:51:42.728256 3341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:51:42.728531 3341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:51:42.746555 3341 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33225
I20260929 15:51:42.746593 3407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33225 every 8 connection(s)
I20260929 15:51:42.747881 3341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:51:42.751549 3408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:42.757122 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 3341
I20260929 15:51:42.757333 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:51:42.758008 388 external_mini_cluster.cc:1428] Setting key 5dad6a351c1f2d96b63e39e71d7dff60
I20260929 15:51:42.762199 3408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0d8b46c768746fbabaaeaf8f739db3c: Bootstrap starting.
I20260929 15:51:42.766098 3408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0d8b46c768746fbabaaeaf8f739db3c: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:42.767238 3408 log.cc:824] T 00000000000000000000000000000000 P d0d8b46c768746fbabaaeaf8f739db3c: Log is configured to *not* fsync() on all Append() calls
I20260929 15:51:42.770190 3408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0d8b46c768746fbabaaeaf8f739db3c: No bootstrap required, opened a new log
Sep 29 15:51:42 dist-test-slave-r0ck krb5kdc[3328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:51:42.778128 3408 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0d8b46c768746fbabaaeaf8f739db3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d8b46c768746fbabaaeaf8f739db3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33225 } }
I20260929 15:51:42.778532 3408 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0d8b46c768746fbabaaeaf8f739db3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:42.778667 3408 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0d8b46c768746fbabaaeaf8f739db3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0d8b46c768746fbabaaeaf8f739db3c, State: Initialized, Role: FOLLOWER
I20260929 15:51:42.779402 3408 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0d8b46c768746fbabaaeaf8f739db3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d8b46c768746fbabaaeaf8f739db3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33225 } }
I20260929 15:51:42.779662 3408 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0d8b46c768746fbabaaeaf8f739db3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:51:42.779822 3408 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0d8b46c768746fbabaaeaf8f739db3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:51:42.780030 3408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0d8b46c768746fbabaaeaf8f739db3c [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:42.780622 3411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:42.760036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56186 (local address 127.0.97.62:33225)
0929 15:51:42.760828 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:42.760840 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:42.760881 (+ 41us) server_negotiation.cc:407] Connection header received
0929 15:51:42.762636 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:42.762660 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:42.763063 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:42.763375 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:42.765295 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:42.766700 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:42.768392 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:42.769064 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:42.769253 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:42.773382 (+ 4129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:42.773398 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:42.773424 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:42.773470 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:42.776413 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:42.777101 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:42.777112 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:42.777125 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:42.777285 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:42.777685 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:42.777691 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:42.777694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:42.778218 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:42.778502 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:42.778936 (+ 434us) server_negotiation.cc:299] Negotiation successful
0929 15:51:42.779359 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":383,"threads_started":1}
I20260929 15:51:42.782990 3408 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0d8b46c768746fbabaaeaf8f739db3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d8b46c768746fbabaaeaf8f739db3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33225 } }
I20260929 15:51:42.783571 3408 leader_election.cc:302] T 00000000000000000000000000000000 P d0d8b46c768746fbabaaeaf8f739db3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0d8b46c768746fbabaaeaf8f739db3c; no voters:
I20260929 15:51:42.784267 3408 leader_election.cc:288] T 00000000000000000000000000000000 P d0d8b46c768746fbabaaeaf8f739db3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:51:42.784391 3413 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0d8b46c768746fbabaaeaf8f739db3c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:51:42.784943 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:51:42.786391 3408 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0d8b46c768746fbabaaeaf8f739db3c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:51:42.790516 3413 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0d8b46c768746fbabaaeaf8f739db3c [term 1 LEADER]: Becoming Leader. State: Replica: d0d8b46c768746fbabaaeaf8f739db3c, State: Running, Role: LEADER
I20260929 15:51:42.791401 3413 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0d8b46c768746fbabaaeaf8f739db3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d8b46c768746fbabaaeaf8f739db3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33225 } }
I20260929 15:51:42.795482 3415 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0d8b46c768746fbabaaeaf8f739db3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0d8b46c768746fbabaaeaf8f739db3c. Latest consensus state: current_term: 1 leader_uuid: "d0d8b46c768746fbabaaeaf8f739db3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d8b46c768746fbabaaeaf8f739db3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33225 } } }
I20260929 15:51:42.795557 3414 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0d8b46c768746fbabaaeaf8f739db3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0d8b46c768746fbabaaeaf8f739db3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d8b46c768746fbabaaeaf8f739db3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33225 } } }
I20260929 15:51:42.796205 3414 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0d8b46c768746fbabaaeaf8f739db3c [sys.catalog]: This master's current role is: LEADER
I20260929 15:51:42.796206 3415 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0d8b46c768746fbabaaeaf8f739db3c [sys.catalog]: This master's current role is: LEADER
W20260929 15:51:42.801055 3428 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d0d8b46c768746fbabaaeaf8f739db3c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 15:51:42.801186 3428 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 15:51:42.801324 3429 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:51:42.803673 3429 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:51:42.812327 3429 catalog_manager.cc:1347] Generated new cluster ID: 2cadf48c8c734024b55b8e1483d27379
I20260929 15:51:42.812453 3429 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:51:42.825556 3429 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:51:42.826905 3429 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:51:42.832808 3411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:42.823474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56194 (local address 127.0.97.62:33225)
0929 15:51:42.823708 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:42.823717 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:42.824300 (+ 583us) server_negotiation.cc:407] Connection header received
0929 15:51:42.824494 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:42.824500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:42.824589 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:42.824709 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:42.825954 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:42.827157 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:42.828985 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:42.829667 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:42.829751 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:42.831095 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:42.831102 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:42.831107 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:51:42.831936 (+ 829us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:42.832015 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:51:42.832042 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:42.832156 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:42.832294 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 15:51:42.832427 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":73}
I20260929 15:51:42.846413 3429 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0d8b46c768746fbabaaeaf8f739db3c: Generated new TSK 0
I20260929 15:51:42.847482 3429 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:51:42.856415 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 3341
I20260929 15:51:42.857666 3403 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:51:42.947896 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 3341
2026-09-29T15:51:42Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1880 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:51:43 dist-test-slave-r0ck krb5kdc[3442](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:51:43 dist-test-slave-r0ck krb5kdc[3442](info): set up 2 sockets
Sep 29 15:51:43 dist-test-slave-r0ck krb5kdc[3442](info): commencing operation
krb5kdc: starting...
W20260929 15:51:44.029717 388 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:51:44 dist-test-slave-r0ck krb5kdc[3442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:51:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:51:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:51:44.217130 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33395
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:51:44.528322 3455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:51:44.528658 3455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:51:44.528846 3455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:51:44.536017 3455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:51:44.536154 3455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:51:44.536260 3455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:51:44.536368 3455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:51:44.536455 3455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:51:44.545929 3455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:51:44.547144 3455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:51:44.548390 3455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:51:44.559623 3460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:44.559547 3461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:44.559585 3463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:51:44.560580 3455 server_base.cc:1034] running on GCE node
I20260929 15:51:44.561254 3455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:51:44.562604 3455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:51:44.563825 3455 hybrid_clock.cc:648] HybridClock initialized: now 1790697104563764 us; error 36 us; skew 500 ppm
I20260929 15:51:44.564186 3455 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:51:44 dist-test-slave-r0ck krb5kdc[3442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:51:44.570248 3455 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:51:44.571811 3455 webserver.cc:460] Webserver started at http://127.0.97.62:37495/ using document root <none> and password file <none>
I20260929 15:51:44.572448 3455 fs_manager.cc:359] Metadata directory not provided
I20260929 15:51:44.572544 3455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:51:44.572782 3455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:51:44.575685 3455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "f3f555a411cf49be8aecb0cc50c176b5"
format_stamp: "Formatted at 2026-09-29 15:51:44 on dist-test-slave-r0ck"
server_key: "cdd7f52ca416604b3be2eb698ea9a577"
server_key_iv: "c3988d594b8b6a61760dcb7268389b6e"
server_key_version: "encryptionkey@0"
I20260929 15:51:44.576385 3455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "f3f555a411cf49be8aecb0cc50c176b5"
format_stamp: "Formatted at 2026-09-29 15:51:44 on dist-test-slave-r0ck"
server_key: "cdd7f52ca416604b3be2eb698ea9a577"
server_key_iv: "c3988d594b8b6a61760dcb7268389b6e"
server_key_version: "encryptionkey@0"
I20260929 15:51:44.581221 3455 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 15:51:44.584501 3470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:44.585830 3455 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:51:44.586005 3455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "f3f555a411cf49be8aecb0cc50c176b5"
format_stamp: "Formatted at 2026-09-29 15:51:44 on dist-test-slave-r0ck"
server_key: "cdd7f52ca416604b3be2eb698ea9a577"
server_key_iv: "c3988d594b8b6a61760dcb7268389b6e"
server_key_version: "encryptionkey@0"
I20260929 15:51:44.586184 3455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:51:44.608038 3455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:51:44.612141 3455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:51:44.612417 3455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:51:44.630124 3455 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33395
I20260929 15:51:44.630162 3521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33395 every 8 connection(s)
I20260929 15:51:44.631351 3455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:51:44.633373 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 3455
I20260929 15:51:44.633589 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:51:44.634438 388 external_mini_cluster.cc:1428] Setting key e7fddf068e3c4a6111c8c143a4838f5d
I20260929 15:51:44.635672 3522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:44.646873 3522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3f555a411cf49be8aecb0cc50c176b5: Bootstrap starting.
Sep 29 15:51:44 dist-test-slave-r0ck krb5kdc[3442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:51:44.649833 3522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3f555a411cf49be8aecb0cc50c176b5: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:44.650820 3522 log.cc:824] T 00000000000000000000000000000000 P f3f555a411cf49be8aecb0cc50c176b5: Log is configured to *not* fsync() on all Append() calls
I20260929 15:51:44.653532 3522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3f555a411cf49be8aecb0cc50c176b5: No bootstrap required, opened a new log
I20260929 15:51:44.655037 3525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:44.636513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43474 (local address 127.0.97.62:33395)
0929 15:51:44.637271 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:44.637282 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:44.637323 (+ 41us) server_negotiation.cc:407] Connection header received
0929 15:51:44.638875 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:44.638895 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:44.639251 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:44.639654 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:44.641012 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:44.642347 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:44.643545 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:44.644225 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:44.644425 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:44.648284 (+ 3859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:44.648291 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:44.648309 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:44.648353 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:44.651357 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:44.651976 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:44.651990 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:44.651996 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:44.652142 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:44.652507 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:44.652513 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:44.652516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:44.653001 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:44.653292 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:44.653742 (+ 450us) server_negotiation.cc:299] Negotiation successful
0929 15:51:44.654140 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":339,"threads_started":1}
I20260929 15:51:44.661224 3522 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3f555a411cf49be8aecb0cc50c176b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f555a411cf49be8aecb0cc50c176b5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33395 } }
I20260929 15:51:44.661649 3522 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3f555a411cf49be8aecb0cc50c176b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:44.661764 3522 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3f555a411cf49be8aecb0cc50c176b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3f555a411cf49be8aecb0cc50c176b5, State: Initialized, Role: FOLLOWER
I20260929 15:51:44.662281 3522 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3f555a411cf49be8aecb0cc50c176b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f555a411cf49be8aecb0cc50c176b5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33395 } }
I20260929 15:51:44.662510 3522 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3f555a411cf49be8aecb0cc50c176b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:51:44.662657 3522 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3f555a411cf49be8aecb0cc50c176b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:51:44.662837 3522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3f555a411cf49be8aecb0cc50c176b5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:44.664947 3522 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3f555a411cf49be8aecb0cc50c176b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f555a411cf49be8aecb0cc50c176b5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33395 } }
I20260929 15:51:44.665582 3522 leader_election.cc:302] T 00000000000000000000000000000000 P f3f555a411cf49be8aecb0cc50c176b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3f555a411cf49be8aecb0cc50c176b5; no voters:
I20260929 15:51:44.666220 3522 leader_election.cc:288] T 00000000000000000000000000000000 P f3f555a411cf49be8aecb0cc50c176b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:51:44.666335 3527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3f555a411cf49be8aecb0cc50c176b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:51:44.667093 3527 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3f555a411cf49be8aecb0cc50c176b5 [term 1 LEADER]: Becoming Leader. State: Replica: f3f555a411cf49be8aecb0cc50c176b5, State: Running, Role: LEADER
I20260929 15:51:44.667644 3527 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3f555a411cf49be8aecb0cc50c176b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f555a411cf49be8aecb0cc50c176b5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33395 } }
I20260929 15:51:44.668569 3522 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3f555a411cf49be8aecb0cc50c176b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:51:44.670627 3529 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3f555a411cf49be8aecb0cc50c176b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3f555a411cf49be8aecb0cc50c176b5. Latest consensus state: current_term: 1 leader_uuid: "f3f555a411cf49be8aecb0cc50c176b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f555a411cf49be8aecb0cc50c176b5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33395 } } }
I20260929 15:51:44.670810 3528 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3f555a411cf49be8aecb0cc50c176b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3f555a411cf49be8aecb0cc50c176b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f555a411cf49be8aecb0cc50c176b5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33395 } } }
I20260929 15:51:44.671077 3529 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3f555a411cf49be8aecb0cc50c176b5 [sys.catalog]: This master's current role is: LEADER
I20260929 15:51:44.671150 3528 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3f555a411cf49be8aecb0cc50c176b5 [sys.catalog]: This master's current role is: LEADER
I20260929 15:51:44.673075 3533 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:51:44.677791 3533 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:51:44.686290 3533 catalog_manager.cc:1347] Generated new cluster ID: dc8d99fe88a7409591b817823bfac3ec
I20260929 15:51:44.686403 3533 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:51:44.708881 3533 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:51:44.710772 3533 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:51:44.721915 3533 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3f555a411cf49be8aecb0cc50c176b5: Generated new TSK 0
I20260929 15:51:44.722909 3533 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 15:51:44.768265 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33395
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:51:45.080475 3548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:51:45.080729 3548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:51:45.080888 3548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:51:45.087905 3548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:51:45.088001 3548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:51:45.088171 3548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:51:45.097735 3548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:51:45.099020 3548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:51:45.100329 3548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:51:45.113032 3556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:45.113124 3554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:45.113101 3553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:51:45.114485 3548 server_base.cc:1034] running on GCE node
I20260929 15:51:45.115168 3548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:51:45.116156 3548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:51:45.117447 3548 hybrid_clock.cc:648] HybridClock initialized: now 1790697105117332 us; error 37 us; skew 500 ppm
I20260929 15:51:45.117815 3548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:51:45 dist-test-slave-r0ck krb5kdc[3442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:51:45.121078 3548 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:51:45.122682 3548 webserver.cc:460] Webserver started at http://127.0.97.1:36521/ using document root <none> and password file <none>
I20260929 15:51:45.123383 3548 fs_manager.cc:359] Metadata directory not provided
I20260929 15:51:45.123528 3548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:51:45.123823 3548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:51:45.126694 3548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-0/data/instance:
uuid: "ec50dc76e8dc49e69e8b71cfc7c8eed8"
format_stamp: "Formatted at 2026-09-29 15:51:45 on dist-test-slave-r0ck"
server_key: "6da59ebdf1b45c7b1dd1f7c26a4b776c"
server_key_iv: "01f3615bfd9bf6ab30a0b3c616eb5e3a"
server_key_version: "encryptionkey@0"
I20260929 15:51:45.127453 3548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ec50dc76e8dc49e69e8b71cfc7c8eed8"
format_stamp: "Formatted at 2026-09-29 15:51:45 on dist-test-slave-r0ck"
server_key: "6da59ebdf1b45c7b1dd1f7c26a4b776c"
server_key_iv: "01f3615bfd9bf6ab30a0b3c616eb5e3a"
server_key_version: "encryptionkey@0"
I20260929 15:51:45.132369 3548 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 15:51:45.135812 3563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:45.137164 3548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 15:51:45.137421 3548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-0/wal
uuid: "ec50dc76e8dc49e69e8b71cfc7c8eed8"
format_stamp: "Formatted at 2026-09-29 15:51:45 on dist-test-slave-r0ck"
server_key: "6da59ebdf1b45c7b1dd1f7c26a4b776c"
server_key_iv: "01f3615bfd9bf6ab30a0b3c616eb5e3a"
server_key_version: "encryptionkey@0"
I20260929 15:51:45.137632 3548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:51:45.164680 3548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:51:45.168979 3548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:51:45.169273 3548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:51:45.170341 3548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:51:45.171623 3548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:51:45.171725 3548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:45.171864 3548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:51:45.171938 3548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:45.204998 3548 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43175
I20260929 15:51:45.205083 3675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43175 every 8 connection(s)
I20260929 15:51:45.206235 3548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:51:45.206892 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 3548
I20260929 15:51:45.207111 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:51:45.207964 388 external_mini_cluster.cc:1428] Setting key 478fb497db9e765137fbdde840615d46
Sep 29 15:51:45 dist-test-slave-r0ck krb5kdc[3442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 15:51:45.228196 3678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:45.208824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44163 (local address 127.0.97.62:33395)
0929 15:51:45.209270 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:45.209275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:45.210448 (+ 1173us) server_negotiation.cc:407] Connection header received
0929 15:51:45.212205 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:45.212214 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:45.212304 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:45.212426 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:45.213924 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:45.215055 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:45.217726 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:45.218145 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:45.218250 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:45.222614 (+ 4364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:45.222619 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:45.222623 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:45.222656 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:45.225201 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:45.225964 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:45.225969 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:45.225972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:45.226084 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:45.226516 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:45.226521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:45.226525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:45.226849 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:45.227002 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:45.227744 (+ 742us) server_negotiation.cc:299] Negotiation successful
0929 15:51:45.227934 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":243,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 15:51:45.229449 3679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:45.209459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33395 (local address 127.0.97.1:44163)
0929 15:51:45.210274 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:45.210326 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:45.211944 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:45.212618 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:45.212627 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:45.213108 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:51:45.213747 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:45.213765 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:45.215221 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:45.215230 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:45.217536 (+ 2306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:45.217554 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:45.218408 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:45.218423 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:45.218761 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:45.219531 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:51:45.219565 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:51:45.222355 (+ 2790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:51:45.225376 (+ 3021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:45.225386 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:45.225403 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:45.225823 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:45.226198 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:45.226204 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:45.226207 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:45.226396 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:45.226976 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:51:45.226984 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:51:45.227305 (+ 321us) client_negotiation.cc:769] Sending connection context
0929 15:51:45.227642 (+ 337us) client_negotiation.cc:240] Negotiation successful
0929 15:51:45.228059 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":351,"threads_started":1}
I20260929 15:51:45.231070 3676 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33395
I20260929 15:51:45.231386 3676 heartbeater.cc:464] Registering TS with master...
I20260929 15:51:45.232108 3676 heartbeater.cc:511] Master 127.0.97.62:33395 requested a full tablet report, sending...
I20260929 15:51:45.234182 3487 ts_manager.cc:194] Registered new tserver with Master: ec50dc76e8dc49e69e8b71cfc7c8eed8 (127.0.97.1:43175)
I20260929 15:51:45.235621 3487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:44163
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 15:51:45.254665 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33395
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:51:45.571694 3682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:51:45.571978 3682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:51:45.572126 3682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:51:45.581233 3682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:51:45.581406 3682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:51:45.581625 3682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:51:45.592847 3682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:51:45.594208 3682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:51:45.595477 3682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:51:45.609100 3688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:45.609290 3690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:45.609297 3687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:51:45.611626 3682 server_base.cc:1034] running on GCE node
I20260929 15:51:45.612278 3682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:51:45.613234 3682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:51:45.614459 3682 hybrid_clock.cc:648] HybridClock initialized: now 1790697105614398 us; error 61 us; skew 500 ppm
I20260929 15:51:45.614769 3682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:51:45 dist-test-slave-r0ck krb5kdc[3442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:51:45.617939 3682 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:51:45.619241 3682 webserver.cc:460] Webserver started at http://127.0.97.2:42597/ using document root <none> and password file <none>
I20260929 15:51:45.619916 3682 fs_manager.cc:359] Metadata directory not provided
I20260929 15:51:45.620012 3682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:51:45.620237 3682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:51:45.623122 3682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-1/data/instance:
uuid: "696b3a66ac19457da3e2b2432d5e1070"
format_stamp: "Formatted at 2026-09-29 15:51:45 on dist-test-slave-r0ck"
server_key: "eee8636b1d2997707630a1a8cdb4c3d8"
server_key_iv: "92cfabf358d45b44382084d7ea847ed8"
server_key_version: "encryptionkey@0"
I20260929 15:51:45.624012 3682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "696b3a66ac19457da3e2b2432d5e1070"
format_stamp: "Formatted at 2026-09-29 15:51:45 on dist-test-slave-r0ck"
server_key: "eee8636b1d2997707630a1a8cdb4c3d8"
server_key_iv: "92cfabf358d45b44382084d7ea847ed8"
server_key_version: "encryptionkey@0"
I20260929 15:51:45.629302 3682 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260929 15:51:45.633036 3697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:45.634455 3682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 15:51:45.634652 3682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-1/wal
uuid: "696b3a66ac19457da3e2b2432d5e1070"
format_stamp: "Formatted at 2026-09-29 15:51:45 on dist-test-slave-r0ck"
server_key: "eee8636b1d2997707630a1a8cdb4c3d8"
server_key_iv: "92cfabf358d45b44382084d7ea847ed8"
server_key_version: "encryptionkey@0"
I20260929 15:51:45.634853 3682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:51:45.662573 3682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:51:45.667193 3682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:51:45.667603 3682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:51:45.668622 3682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:51:45.670150 3682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:51:45.670248 3682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:45.670397 3682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:51:45.670467 3682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:45.704384 3682 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:32799
I20260929 15:51:45.704466 3809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:32799 every 8 connection(s)
I20260929 15:51:45.705628 3682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:51:45.712855 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 3682
I20260929 15:51:45.713052 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:51:45.713690 388 external_mini_cluster.cc:1428] Setting key c4c249413703bd5a5c1a8b82e79ee9f2
Sep 29 15:51:45 dist-test-slave-r0ck krb5kdc[3442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:51:45.728263 3678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:45.708143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60065 (local address 127.0.97.62:33395)
0929 15:51:45.708342 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:45.708346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:45.709514 (+ 1168us) server_negotiation.cc:407] Connection header received
0929 15:51:45.711617 (+ 2103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:45.711623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:45.711736 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:45.711861 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:45.713978 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:45.714872 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:45.717141 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:45.717855 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:45.717969 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:45.722594 (+ 4625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:45.722605 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:45.722609 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:45.722641 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:45.725214 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:45.725977 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:45.725982 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:45.725992 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:45.726122 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:45.726565 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:45.726571 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:45.726579 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:45.726824 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:45.726990 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:45.727671 (+ 681us) server_negotiation.cc:299] Negotiation successful
0929 15:51:45.727914 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 15:51:45.729291 3812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:45.708575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33395 (local address 127.0.97.2:60065)
0929 15:51:45.709297 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:45.709391 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:45.711306 (+ 1915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:45.712111 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:45.712119 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:45.712879 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:51:45.713758 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:45.713787 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:45.715042 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:45.715048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:45.716903 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:45.716920 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:45.718122 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:45.718132 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:45.718511 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:45.719345 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:51:45.719381 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:51:45.722345 (+ 2964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:51:45.725397 (+ 3052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:45.725405 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:45.725424 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:45.725846 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:45.726241 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:45.726248 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:45.726251 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:45.726439 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:45.726953 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:51:45.726962 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:51:45.727306 (+ 344us) client_negotiation.cc:769] Sending connection context
0929 15:51:45.727582 (+ 276us) client_negotiation.cc:240] Negotiation successful
0929 15:51:45.728132 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":324,"threads_started":1}
I20260929 15:51:45.731240 3810 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33395
I20260929 15:51:45.731761 3810 heartbeater.cc:464] Registering TS with master...
I20260929 15:51:45.732750 3810 heartbeater.cc:511] Master 127.0.97.62:33395 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 15:51:45.734712 3487 ts_manager.cc:194] Registered new tserver with Master: 696b3a66ac19457da3e2b2432d5e1070 (127.0.97.2:32799)
I20260929 15:51:45.735527 3487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:60065
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 15:51:45.761144 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33395
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:51:46.099393 3815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:51:46.099695 3815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:51:46.099869 3815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:51:46.107308 3815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:51:46.107424 3815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:51:46.107604 3815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:51:46.117506 3815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:51:46.118821 3815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:51:46.120106 3815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:51:46.132923 3821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:46.132885 3823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:46.133054 3820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:51:46.134100 3815 server_base.cc:1034] running on GCE node
I20260929 15:51:46.134702 3815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:51:46.135667 3815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:51:46.136981 3815 hybrid_clock.cc:648] HybridClock initialized: now 1790697106136922 us; error 49 us; skew 500 ppm
I20260929 15:51:46.137471 3815 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:51:46 dist-test-slave-r0ck krb5kdc[3442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:51:46.140841 3815 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:51:46.142374 3815 webserver.cc:460] Webserver started at http://127.0.97.3:37519/ using document root <none> and password file <none>
I20260929 15:51:46.143065 3815 fs_manager.cc:359] Metadata directory not provided
I20260929 15:51:46.143211 3815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:51:46.143515 3815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:51:46.146516 3815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-2/data/instance:
uuid: "20c702afa4eb45728f6e1b3f460d8693"
format_stamp: "Formatted at 2026-09-29 15:51:46 on dist-test-slave-r0ck"
server_key: "4d67d9074acb04b43af2e395919be139"
server_key_iv: "7f93153b1286c0a5b1ac7b916b929e35"
server_key_version: "encryptionkey@0"
I20260929 15:51:46.147316 3815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "20c702afa4eb45728f6e1b3f460d8693"
format_stamp: "Formatted at 2026-09-29 15:51:46 on dist-test-slave-r0ck"
server_key: "4d67d9074acb04b43af2e395919be139"
server_key_iv: "7f93153b1286c0a5b1ac7b916b929e35"
server_key_version: "encryptionkey@0"
I20260929 15:51:46.152391 3815 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260929 15:51:46.155851 3830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:46.157255 3815 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 15:51:46.157490 3815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-2/wal
uuid: "20c702afa4eb45728f6e1b3f460d8693"
format_stamp: "Formatted at 2026-09-29 15:51:46 on dist-test-slave-r0ck"
server_key: "4d67d9074acb04b43af2e395919be139"
server_key_iv: "7f93153b1286c0a5b1ac7b916b929e35"
server_key_version: "encryptionkey@0"
I20260929 15:51:46.157708 3815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:51:46.176170 3815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:51:46.180454 3815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:51:46.180800 3815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:51:46.181860 3815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:51:46.183264 3815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:51:46.183380 3815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:46.183532 3815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:51:46.183604 3815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:46.216284 3815 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34781
I20260929 15:51:46.216321 3942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34781 every 8 connection(s)
I20260929 15:51:46.217608 3815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:51:46.221078 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 3815
I20260929 15:51:46.221295 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697075657932-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:51:46.221860 388 external_mini_cluster.cc:1428] Setting key 674df32d60e12e9e10d8c9bfbbb1cb13
Sep 29 15:51:46 dist-test-slave-r0ck krb5kdc[3442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:51:46.237756 3678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:46.220148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48749 (local address 127.0.97.62:33395)
0929 15:51:46.220347 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:46.220351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:46.221555 (+ 1204us) server_negotiation.cc:407] Connection header received
0929 15:51:46.223682 (+ 2127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:46.223694 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:46.223850 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:46.224010 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:46.225679 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:46.226437 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:46.227878 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:46.228253 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:46.228353 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:46.231732 (+ 3379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:46.231737 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:46.231742 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:46.231798 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:46.234401 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:46.235149 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:46.235154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:46.235157 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:46.235315 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:46.235832 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:46.235841 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:46.235843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:46.236268 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:46.236420 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:46.237185 (+ 765us) server_negotiation.cc:299] Negotiation successful
0929 15:51:46.237448 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 15:51:46.238950 3945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:46.220559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33395 (local address 127.0.97.3:48749)
0929 15:51:46.221374 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:46.221426 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:46.223369 (+ 1943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:46.224260 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:46.224273 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:46.224895 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:51:46.225531 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:46.225550 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:46.226568 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:46.226572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:46.227760 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:46.227771 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:46.228431 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:46.228436 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:46.228635 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:46.229267 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:51:46.229293 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:51:46.231515 (+ 2222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:51:46.234539 (+ 3024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:46.234549 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:46.234563 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:46.234989 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:46.235422 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:46.235451 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:46.235457 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:46.235710 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:46.236402 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:51:46.236408 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:51:46.236811 (+ 403us) client_negotiation.cc:769] Sending connection context
0929 15:51:46.237044 (+ 233us) client_negotiation.cc:240] Negotiation successful
0929 15:51:46.237644 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":345,"threads_started":1}
I20260929 15:51:46.238937 3676 heartbeater.cc:503] Master 127.0.97.62:33395 was elected leader, sending a full tablet report...
I20260929 15:51:46.240653 3943 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33395
I20260929 15:51:46.240900 3943 heartbeater.cc:464] Registering TS with master...
I20260929 15:51:46.241724 3943 heartbeater.cc:511] Master 127.0.97.62:33395 requested a full tablet report, sending...
I20260929 15:51:46.242985 3487 ts_manager.cc:194] Registered new tserver with Master: 20c702afa4eb45728f6e1b3f460d8693 (127.0.97.3:34781)
I20260929 15:51:46.243731 3487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48749
I20260929 15:51:46.249552 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 15:51:46 dist-test-slave-r0ck krb5kdc[3442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697106, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 15:51:46 dist-test-slave-r0ck krb5kdc[3442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697106, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:51:46.296240 3678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:46.282344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43476 (local address 127.0.97.62:33395)
0929 15:51:46.282552 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:46.282565 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:46.283075 (+ 510us) server_negotiation.cc:407] Connection header received
0929 15:51:46.283221 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:46.283225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:46.283289 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:46.283383 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:46.284707 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:46.285504 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:46.286744 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:46.287146 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:46.287237 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:46.291175 (+ 3938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:46.291179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:46.291183 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:46.291213 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:46.293704 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:46.294312 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:46.294319 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:46.294327 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:46.294437 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:46.294831 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:46.294837 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:46.294841 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:46.295149 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:46.295399 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:46.295715 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 15:51:46.295923 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260929 15:51:46.296806 3487 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43476
I20260929 15:51:46.298952 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 3548
Sep 29 15:51:46 dist-test-slave-r0ck krb5kdc[3442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697106, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 15:51:46.313535 3954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:46.299935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50774 (local address 127.0.97.1:43175)
0929 15:51:46.300510 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:46.300517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:46.300548 (+ 31us) server_negotiation.cc:407] Connection header received
0929 15:51:46.300601 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:46.300607 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:46.300811 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:46.300929 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:46.302052 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:46.303030 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:46.304129 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:46.304668 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:46.304765 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:46.308398 (+ 3633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:46.308408 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:46.308421 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:46.308453 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:46.311164 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:46.311736 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:46.311742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:46.311744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:46.311871 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:46.312167 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:46.312170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:46.312172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:46.312473 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:46.312688 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:46.312885 (+ 197us) server_negotiation.cc:299] Negotiation successful
0929 15:51:46.313068 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":290,"threads_started":1}
W20260929 15:51:46.314764 3671 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50774
W20260929 15:51:46.315848 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 3548: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:51:46.316054 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 3548
I20260929 15:51:46.331498 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 3682
Sep 29 15:51:46 dist-test-slave-r0ck krb5kdc[3442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697106, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 15:51:46.346768 3955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:46.332372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42486 (local address 127.0.97.2:32799)
0929 15:51:46.332957 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:46.332965 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:46.332988 (+ 23us) server_negotiation.cc:407] Connection header received
0929 15:51:46.333060 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:46.333066 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:46.333255 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:46.333458 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:46.334593 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:46.335566 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:46.336796 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:46.337311 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:46.337468 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:46.341615 (+ 4147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:46.341624 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:46.341637 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:46.341666 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:46.344235 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:46.344862 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:46.344869 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:46.344872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:46.344988 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:46.345329 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:46.345333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:46.345335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:46.345708 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:46.345936 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:46.346200 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 15:51:46.346409 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":313,"threads_started":1}
W20260929 15:51:46.347973 3805 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42486
W20260929 15:51:46.349057 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 3682: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:51:46.349216 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 3682
I20260929 15:51:46.364809 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 3815
Sep 29 15:51:46 dist-test-slave-r0ck krb5kdc[3442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697106, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 15:51:46.382054 3956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:46.365771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43662 (local address 127.0.97.3:34781)
0929 15:51:46.366388 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:46.366395 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:46.366423 (+ 28us) server_negotiation.cc:407] Connection header received
0929 15:51:46.366482 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:46.366488 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:46.366693 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:46.366864 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:46.368164 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:46.369414 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:46.370711 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:46.371252 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:46.371391 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:46.376670 (+ 5279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:46.376680 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:46.376712 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:46.376746 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:46.379567 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:46.380206 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:46.380213 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:46.380215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:46.380326 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:46.380661 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:46.380664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:46.380666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:46.380986 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:46.381198 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:46.381577 (+ 379us) server_negotiation.cc:299] Negotiation successful
0929 15:51:46.381769 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":332,"threads_started":1}
W20260929 15:51:46.383327 3938 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43662
W20260929 15:51:46.384411 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 3815: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:51:46.384577 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 3815
I20260929 15:51:46.399540 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 3455
I20260929 15:51:46.400550 3517 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:51:46.501555 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 3455
2026-09-29T15:51:46Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3559 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:51:46 dist-test-slave-r0ck krb5kdc[3960](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:51:46 dist-test-slave-r0ck krb5kdc[3960](info): set up 2 sockets
Sep 29 15:51:46 dist-test-slave-r0ck krb5kdc[3960](info): commencing operation
krb5kdc: starting...
W20260929 15:51:47.581761 388 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:51:47 dist-test-slave-r0ck krb5kdc[3960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:51:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:51:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:51:47.765457 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33851
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33851
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:51:48.084161 3973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:51:48.084436 3973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:51:48.084611 3973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:51:48.091774 3973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:51:48.091873 3973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:51:48.091956 3973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:51:48.092056 3973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:51:48.092132 3973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:51:48.101771 3973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:51:48.103008 3973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:51:48.104231 3973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:51:48.115412 3978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:48.115484 3979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:48.115450 3981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:51:48.116473 3973 server_base.cc:1034] running on GCE node
I20260929 15:51:48.117269 3973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:51:48.118599 3973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:51:48.119803 3973 hybrid_clock.cc:648] HybridClock initialized: now 1790697108119745 us; error 52 us; skew 500 ppm
I20260929 15:51:48.120167 3973 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:51:48 dist-test-slave-r0ck krb5kdc[3960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:51:48.123229 3973 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:51:48.124673 3973 webserver.cc:460] Webserver started at http://127.0.97.62:35771/ using document root <none> and password file <none>
I20260929 15:51:48.125319 3973 fs_manager.cc:359] Metadata directory not provided
I20260929 15:51:48.125519 3973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:51:48.125820 3973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:51:48.128605 3973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "127110f9f20c49778746f209eae09156"
format_stamp: "Formatted at 2026-09-29 15:51:48 on dist-test-slave-r0ck"
server_key: "f024c107377d96a0cca427a50b0bfbb2"
server_key_iv: "65d3359cf6a9bfdda0dbae81a1366e4d"
server_key_version: "encryptionkey@0"
I20260929 15:51:48.129400 3973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "127110f9f20c49778746f209eae09156"
format_stamp: "Formatted at 2026-09-29 15:51:48 on dist-test-slave-r0ck"
server_key: "f024c107377d96a0cca427a50b0bfbb2"
server_key_iv: "65d3359cf6a9bfdda0dbae81a1366e4d"
server_key_version: "encryptionkey@0"
I20260929 15:51:48.134349 3973 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 15:51:48.137707 3988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:48.138991 3973 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 15:51:48.139169 3973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "127110f9f20c49778746f209eae09156"
format_stamp: "Formatted at 2026-09-29 15:51:48 on dist-test-slave-r0ck"
server_key: "f024c107377d96a0cca427a50b0bfbb2"
server_key_iv: "65d3359cf6a9bfdda0dbae81a1366e4d"
server_key_version: "encryptionkey@0"
I20260929 15:51:48.139364 3973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:51:48.159410 3973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:51:48.163228 3973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:51:48.163513 3973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:51:48.181190 3973 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33851
I20260929 15:51:48.181222 4039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33851 every 8 connection(s)
I20260929 15:51:48.182507 3973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:51:48.186085 4040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:48.192624 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 3973
I20260929 15:51:48.192852 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:51:48.193638 388 external_mini_cluster.cc:1428] Setting key da0eeb2d1d57bc8ae68e0d8f2121d198
I20260929 15:51:48.197011 4040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 127110f9f20c49778746f209eae09156: Bootstrap starting.
I20260929 15:51:48.200490 4040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 127110f9f20c49778746f209eae09156: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:48.201659 4040 log.cc:824] T 00000000000000000000000000000000 P 127110f9f20c49778746f209eae09156: Log is configured to *not* fsync() on all Append() calls
I20260929 15:51:48.204523 4040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 127110f9f20c49778746f209eae09156: No bootstrap required, opened a new log
Sep 29 15:51:48 dist-test-slave-r0ck krb5kdc[3960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:51:48.212822 4040 raft_consensus.cc:348] T 00000000000000000000000000000000 P 127110f9f20c49778746f209eae09156 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127110f9f20c49778746f209eae09156" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33851 } }
I20260929 15:51:48.213193 4040 raft_consensus.cc:374] T 00000000000000000000000000000000 P 127110f9f20c49778746f209eae09156 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:48.213316 4040 raft_consensus.cc:729] T 00000000000000000000000000000000 P 127110f9f20c49778746f209eae09156 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 127110f9f20c49778746f209eae09156, State: Initialized, Role: FOLLOWER
I20260929 15:51:48.213908 4040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 127110f9f20c49778746f209eae09156 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127110f9f20c49778746f209eae09156" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33851 } }
I20260929 15:51:48.214161 4040 raft_consensus.cc:388] T 00000000000000000000000000000000 P 127110f9f20c49778746f209eae09156 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:51:48.214300 4040 raft_consensus.cc:482] T 00000000000000000000000000000000 P 127110f9f20c49778746f209eae09156 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:51:48.214491 4040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 127110f9f20c49778746f209eae09156 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:48.215864 4043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:48.195629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59576 (local address 127.0.97.62:33851)
0929 15:51:48.196368 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:48.196379 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:48.196416 (+ 37us) server_negotiation.cc:407] Connection header received
0929 15:51:48.198173 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:48.198193 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:48.198546 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:48.198863 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:48.200633 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:48.202016 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:48.203608 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:48.204248 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:48.204381 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:48.208722 (+ 4341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:48.208729 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:48.208742 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:48.208787 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:48.211899 (+ 3112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:48.212538 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:48.212547 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:48.212561 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:48.212706 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:48.213069 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:48.213076 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:48.213079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:48.213787 (+ 708us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:48.214110 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:48.214501 (+ 391us) server_negotiation.cc:299] Negotiation successful
0929 15:51:48.214887 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":353,"threads_started":1}
I20260929 15:51:48.216847 4040 raft_consensus.cc:504] T 00000000000000000000000000000000 P 127110f9f20c49778746f209eae09156 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127110f9f20c49778746f209eae09156" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33851 } }
I20260929 15:51:48.217326 4040 leader_election.cc:302] T 00000000000000000000000000000000 P 127110f9f20c49778746f209eae09156 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 127110f9f20c49778746f209eae09156; no voters:
I20260929 15:51:48.218086 4040 leader_election.cc:288] T 00000000000000000000000000000000 P 127110f9f20c49778746f209eae09156 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:51:48.218196 4045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 127110f9f20c49778746f209eae09156 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:51:48.218835 4045 raft_consensus.cc:686] T 00000000000000000000000000000000 P 127110f9f20c49778746f209eae09156 [term 1 LEADER]: Becoming Leader. State: Replica: 127110f9f20c49778746f209eae09156, State: Running, Role: LEADER
I20260929 15:51:48.219492 4045 consensus_queue.cc:237] T 00000000000000000000000000000000 P 127110f9f20c49778746f209eae09156 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127110f9f20c49778746f209eae09156" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33851 } }
I20260929 15:51:48.221555 4045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 127110f9f20c49778746f209eae09156 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 127110f9f20c49778746f209eae09156. Latest consensus state: current_term: 1 leader_uuid: "127110f9f20c49778746f209eae09156" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127110f9f20c49778746f209eae09156" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33851 } } }
I20260929 15:51:48.222110 4045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 127110f9f20c49778746f209eae09156 [sys.catalog]: This master's current role is: LEADER
I20260929 15:51:48.223917 4047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 127110f9f20c49778746f209eae09156 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "127110f9f20c49778746f209eae09156" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127110f9f20c49778746f209eae09156" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33851 } } }
I20260929 15:51:48.224903 4047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 127110f9f20c49778746f209eae09156 [sys.catalog]: This master's current role is: LEADER
I20260929 15:51:48.225055 4040 sys_catalog.cc:564] T 00000000000000000000000000000000 P 127110f9f20c49778746f209eae09156 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:51:48.230687 4049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:51:48.235229 4049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:51:48.243134 4049 catalog_manager.cc:1347] Generated new cluster ID: e1015bdc2d104c0aa8d3541aa7ab62f3
I20260929 15:51:48.243350 4049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:51:48.259958 4049 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:51:48.261266 4049 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:51:48.274289 4049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 127110f9f20c49778746f209eae09156: Generated new TSK 0
I20260929 15:51:48.275336 4049 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 15:51:48.327069 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33851
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:51:48.642544 4066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:51:48.642894 4066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:51:48.643064 4066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:51:48.650425 4066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:51:48.650522 4066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:51:48.650727 4066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:51:48.660230 4066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:51:48.661470 4066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:51:48.662708 4066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:51:48.675576 4071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:48.675625 4072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:48.676260 4074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:51:48.676689 4066 server_base.cc:1034] running on GCE node
I20260929 15:51:48.677263 4066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:51:48.678172 4066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:51:48.679399 4066 hybrid_clock.cc:648] HybridClock initialized: now 1790697108679343 us; error 68 us; skew 500 ppm
I20260929 15:51:48.679703 4066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:51:48 dist-test-slave-r0ck krb5kdc[3960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:51:48.682703 4066 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:51:48.684139 4066 webserver.cc:460] Webserver started at http://127.0.97.1:37891/ using document root <none> and password file <none>
I20260929 15:51:48.684787 4066 fs_manager.cc:359] Metadata directory not provided
I20260929 15:51:48.684927 4066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:51:48.685227 4066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:51:48.688050 4066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-0/data/instance:
uuid: "68d6c7efe38b4b92b7b3589dc89cefb0"
format_stamp: "Formatted at 2026-09-29 15:51:48 on dist-test-slave-r0ck"
server_key: "55732b71f36b072df7cd4a2266bbaa29"
server_key_iv: "8704c7a55e275ecc70c29d4c716aa5ed"
server_key_version: "encryptionkey@0"
I20260929 15:51:48.688824 4066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "68d6c7efe38b4b92b7b3589dc89cefb0"
format_stamp: "Formatted at 2026-09-29 15:51:48 on dist-test-slave-r0ck"
server_key: "55732b71f36b072df7cd4a2266bbaa29"
server_key_iv: "8704c7a55e275ecc70c29d4c716aa5ed"
server_key_version: "encryptionkey@0"
I20260929 15:51:48.693634 4066 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.006s
I20260929 15:51:48.696869 4081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:48.698171 4066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:51:48.698346 4066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-0/wal
uuid: "68d6c7efe38b4b92b7b3589dc89cefb0"
format_stamp: "Formatted at 2026-09-29 15:51:48 on dist-test-slave-r0ck"
server_key: "55732b71f36b072df7cd4a2266bbaa29"
server_key_iv: "8704c7a55e275ecc70c29d4c716aa5ed"
server_key_version: "encryptionkey@0"
I20260929 15:51:48.698539 4066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:51:48.722708 4066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:51:48.726931 4066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:51:48.727275 4066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:51:48.728261 4066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:51:48.729696 4066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:51:48.729768 4066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:48.729847 4066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:51:48.729890 4066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:48.761832 4066 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34265
I20260929 15:51:48.761930 4193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34265 every 8 connection(s)
I20260929 15:51:48.763077 4066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:51:48.765592 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 4066
I20260929 15:51:48.765832 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:51:48.766587 388 external_mini_cluster.cc:1428] Setting key 7f59015bd9412d07dde760084c918003
Sep 29 15:51:48 dist-test-slave-r0ck krb5kdc[3960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:51:48.783651 4196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:48.765537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53705 (local address 127.0.97.62:33851)
0929 15:51:48.766068 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:48.766075 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:48.767070 (+ 995us) server_negotiation.cc:407] Connection header received
0929 15:51:48.769011 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:48.769015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:48.769075 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:48.769162 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:48.771202 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:48.771884 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:48.774399 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:48.774833 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:48.774970 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:48.778284 (+ 3314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:48.778289 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:48.778292 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:48.778321 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:48.780734 (+ 2413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:48.781497 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:48.781501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:48.781503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:48.781607 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:48.782029 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:48.782033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:48.782037 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:48.782287 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:48.782445 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:48.783080 (+ 635us) server_negotiation.cc:299] Negotiation successful
0929 15:51:48.783295 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":277,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 15:51:48.784554 4197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:48.766082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33851 (local address 127.0.97.1:53705)
0929 15:51:48.766895 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:48.766946 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:48.768711 (+ 1765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:48.769448 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:48.769461 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:48.770229 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:51:48.771033 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:48.771059 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:48.772034 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:48.772042 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:48.774208 (+ 2166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:48.774230 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:48.775050 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:48.775056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:48.775284 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:48.775955 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:51:48.775976 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:51:48.778058 (+ 2082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:51:48.780897 (+ 2839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:48.780905 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:48.780920 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:48.781323 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:48.781715 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:48.781721 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:48.781724 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:48.781914 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:48.782424 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:51:48.782432 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:51:48.782732 (+ 300us) client_negotiation.cc:769] Sending connection context
0929 15:51:48.782982 (+ 250us) client_negotiation.cc:240] Negotiation successful
0929 15:51:48.783456 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":309,"threads_started":1}
I20260929 15:51:48.786070 4194 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33851
I20260929 15:51:48.786396 4194 heartbeater.cc:464] Registering TS with master...
I20260929 15:51:48.787081 4194 heartbeater.cc:511] Master 127.0.97.62:33851 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 15:51:48.789108 4005 ts_manager.cc:194] Registered new tserver with Master: 68d6c7efe38b4b92b7b3589dc89cefb0 (127.0.97.1:34265)
I20260929 15:51:48.790482 4005 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53705
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 15:51:48.809566 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33851
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:51:49.137753 4200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:51:49.138054 4200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:51:49.138230 4200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:51:49.145637 4200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:51:49.145772 4200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:51:49.145963 4200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:51:49.155433 4200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:51:49.156708 4200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:51:49.158018 4200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:51:49.171159 4205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:49.171275 4206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:49.171231 4208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:51:49.172027 4200 server_base.cc:1034] running on GCE node
I20260929 15:51:49.172610 4200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:51:49.173560 4200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:51:49.174778 4200 hybrid_clock.cc:648] HybridClock initialized: now 1790697109174739 us; error 30 us; skew 500 ppm
I20260929 15:51:49.175087 4200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:51:49 dist-test-slave-r0ck krb5kdc[3960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:51:49.178076 4200 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:51:49.179448 4200 webserver.cc:460] Webserver started at http://127.0.97.2:46157/ using document root <none> and password file <none>
I20260929 15:51:49.180104 4200 fs_manager.cc:359] Metadata directory not provided
I20260929 15:51:49.180246 4200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:51:49.180621 4200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:51:49.184250 4200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-1/data/instance:
uuid: "bf342b5afa93438b91a43910439a0d48"
format_stamp: "Formatted at 2026-09-29 15:51:49 on dist-test-slave-r0ck"
server_key: "39bb298d72110115675edba013d873cf"
server_key_iv: "3350750ef99b9bf5419355dd049061c7"
server_key_version: "encryptionkey@0"
I20260929 15:51:49.185168 4200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "bf342b5afa93438b91a43910439a0d48"
format_stamp: "Formatted at 2026-09-29 15:51:49 on dist-test-slave-r0ck"
server_key: "39bb298d72110115675edba013d873cf"
server_key_iv: "3350750ef99b9bf5419355dd049061c7"
server_key_version: "encryptionkey@0"
I20260929 15:51:49.191202 4200 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260929 15:51:49.195159 4215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:49.196647 4200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 15:51:49.196795 4200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-1/wal
uuid: "bf342b5afa93438b91a43910439a0d48"
format_stamp: "Formatted at 2026-09-29 15:51:49 on dist-test-slave-r0ck"
server_key: "39bb298d72110115675edba013d873cf"
server_key_iv: "3350750ef99b9bf5419355dd049061c7"
server_key_version: "encryptionkey@0"
I20260929 15:51:49.196997 4200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:51:49.217868 4200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:51:49.221513 4200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:51:49.221735 4200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:51:49.222586 4200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:51:49.223802 4200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:51:49.223881 4200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:49.223963 4200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:51:49.224006 4200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:49.257622 4200 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33111
I20260929 15:51:49.257665 4327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33111 every 8 connection(s)
I20260929 15:51:49.258962 4200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:51:49.266598 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 4200
I20260929 15:51:49.266791 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:51:49.267184 388 external_mini_cluster.cc:1428] Setting key 139103a7583b2b3f4d74f18a39f259e5
Sep 29 15:51:49 dist-test-slave-r0ck krb5kdc[3960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:51:49.279068 4196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:49.261422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47587 (local address 127.0.97.62:33851)
0929 15:51:49.261601 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:49.261606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:49.262605 (+ 999us) server_negotiation.cc:407] Connection header received
0929 15:51:49.264350 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:49.264355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:49.264443 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:49.264551 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:49.266014 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:49.266811 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:49.268496 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:49.268987 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:49.269115 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:49.272800 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:49.272806 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:49.272809 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:49.272847 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:49.275810 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:49.276734 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:49.276738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:49.276740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:49.276874 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:49.277392 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:49.277397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:49.277403 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:49.277659 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:49.277767 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:49.278669 (+ 902us) server_negotiation.cc:299] Negotiation successful
0929 15:51:49.278837 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 15:51:49.280447 4330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:49.261764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33851 (local address 127.0.97.2:47587)
0929 15:51:49.262444 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:49.262484 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:49.264083 (+ 1599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:49.264780 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:49.264788 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:49.265247 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:51:49.265847 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:49.265864 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:49.266954 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:49.266961 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:49.268349 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:49.268360 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:49.269216 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:49.269223 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:49.269516 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:49.270235 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:51:49.270262 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:51:49.272594 (+ 2332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:51:49.275955 (+ 3361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:49.275965 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:49.275984 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:49.276599 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:49.276987 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:49.276995 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:49.277002 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:49.277224 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:49.277785 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:51:49.277795 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:51:49.278216 (+ 421us) client_negotiation.cc:769] Sending connection context
0929 15:51:49.278527 (+ 311us) client_negotiation.cc:240] Negotiation successful
0929 15:51:49.279057 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":270,"threads_started":1}
I20260929 15:51:49.281863 4328 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33851
I20260929 15:51:49.282168 4328 heartbeater.cc:464] Registering TS with master...
I20260929 15:51:49.283031 4328 heartbeater.cc:511] Master 127.0.97.62:33851 requested a full tablet report, sending...
I20260929 15:51:49.284571 4005 ts_manager.cc:194] Registered new tserver with Master: bf342b5afa93438b91a43910439a0d48 (127.0.97.2:33111)
I20260929 15:51:49.285431 4005 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:47587
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 15:51:49.310470 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33851
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:51:49.621924 4333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:51:49.622180 4333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:51:49.622273 4333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:51:49.629402 4333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:51:49.629495 4333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:51:49.629602 4333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:51:49.638739 4333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:51:49.639824 4333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:51:49.641265 4333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:51:49.655457 4341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:49.655527 4338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:49.655474 4339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:51:49.656286 4333 server_base.cc:1034] running on GCE node
I20260929 15:51:49.656908 4333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:51:49.657872 4333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:51:49.659127 4333 hybrid_clock.cc:648] HybridClock initialized: now 1790697109659075 us; error 51 us; skew 500 ppm
I20260929 15:51:49.659461 4333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:51:49 dist-test-slave-r0ck krb5kdc[3960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:51:49.666958 4333 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:51:49.668538 4333 webserver.cc:460] Webserver started at http://127.0.97.3:36139/ using document root <none> and password file <none>
I20260929 15:51:49.669237 4333 fs_manager.cc:359] Metadata directory not provided
I20260929 15:51:49.669410 4333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:51:49.669720 4333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:51:49.672573 4333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-2/data/instance:
uuid: "1ee74f14f6284d2aa4e463bca1b1f46e"
format_stamp: "Formatted at 2026-09-29 15:51:49 on dist-test-slave-r0ck"
server_key: "7b9459abcb7025bdbb92ad89c73eaa1d"
server_key_iv: "effc41fb2ae15bf24e81934c729bbb66"
server_key_version: "encryptionkey@0"
I20260929 15:51:49.673432 4333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "1ee74f14f6284d2aa4e463bca1b1f46e"
format_stamp: "Formatted at 2026-09-29 15:51:49 on dist-test-slave-r0ck"
server_key: "7b9459abcb7025bdbb92ad89c73eaa1d"
server_key_iv: "effc41fb2ae15bf24e81934c729bbb66"
server_key_version: "encryptionkey@0"
I20260929 15:51:49.678500 4333 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260929 15:51:49.682235 4348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:49.683634 4333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:51:49.683820 4333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-2/wal
uuid: "1ee74f14f6284d2aa4e463bca1b1f46e"
format_stamp: "Formatted at 2026-09-29 15:51:49 on dist-test-slave-r0ck"
server_key: "7b9459abcb7025bdbb92ad89c73eaa1d"
server_key_iv: "effc41fb2ae15bf24e81934c729bbb66"
server_key_version: "encryptionkey@0"
I20260929 15:51:49.684053 4333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:51:49.701220 4333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:51:49.705644 4333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:51:49.705940 4333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:51:49.706884 4333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:51:49.708178 4333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:51:49.708285 4333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:49.708452 4333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:51:49.708540 4333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:49.743731 4333 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46681
I20260929 15:51:49.743763 4460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46681 every 8 connection(s)
I20260929 15:51:49.745208 4333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:51:49.746851 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 4333
I20260929 15:51:49.747043 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697075657932-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:51:49.747534 388 external_mini_cluster.cc:1428] Setting key 51be7381e15a0f9791b887a3ed148037
Sep 29 15:51:49 dist-test-slave-r0ck krb5kdc[3960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:51:49.766026 4196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:49.747977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39033 (local address 127.0.97.62:33851)
0929 15:51:49.748179 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:49.748185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:49.750070 (+ 1885us) server_negotiation.cc:407] Connection header received
0929 15:51:49.751933 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:49.751938 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:49.752022 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:49.752133 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:49.753768 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:49.754535 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:49.756039 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:49.756444 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:49.756579 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:49.760021 (+ 3442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:49.760029 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:49.760039 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:49.760076 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:49.763109 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:49.763802 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:49.763807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:49.763811 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:49.763942 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:49.764369 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:49.764374 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:49.764377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:49.764638 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:49.764788 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:49.765430 (+ 642us) server_negotiation.cc:299] Negotiation successful
0929 15:51:49.765680 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 15:51:49.766957 4463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:49.748549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33851 (local address 127.0.97.3:39033)
0929 15:51:49.749924 (+ 1375us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:49.749966 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:49.751668 (+ 1702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:49.752362 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:49.752370 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:49.752913 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:51:49.753600 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:49.753621 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:49.754695 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:49.754705 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:49.755886 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:49.755897 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:49.756660 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:49.756669 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:49.756914 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:49.757707 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:51:49.757729 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:51:49.759817 (+ 2088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:51:49.763214 (+ 3397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:49.763223 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:49.763237 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:49.763675 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:49.764052 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:49.764058 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:49.764062 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:49.764245 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:49.764762 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:51:49.764769 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:51:49.765062 (+ 293us) client_negotiation.cc:769] Sending connection context
0929 15:51:49.765310 (+ 248us) client_negotiation.cc:240] Negotiation successful
0929 15:51:49.765857 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1104,"thread_start_us":897,"threads_started":1}
I20260929 15:51:49.768278 4461 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33851
I20260929 15:51:49.768554 4461 heartbeater.cc:464] Registering TS with master...
I20260929 15:51:49.769234 4461 heartbeater.cc:511] Master 127.0.97.62:33851 requested a full tablet report, sending...
I20260929 15:51:49.770538 4005 ts_manager.cc:194] Registered new tserver with Master: 1ee74f14f6284d2aa4e463bca1b1f46e (127.0.97.3:46681)
I20260929 15:51:49.771310 4005 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39033
I20260929 15:51:49.775117 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:51:49.793521 4194 heartbeater.cc:503] Master 127.0.97.62:33851 was elected leader, sending a full tablet report...
Sep 29 15:51:49 dist-test-slave-r0ck krb5kdc[3960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 15:51:49 dist-test-slave-r0ck krb5kdc[3960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 15:51:49.825019 4468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:49.810763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32980 (local address 127.0.97.1:34265)
0929 15:51:49.811398 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:49.811405 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:49.811430 (+ 25us) server_negotiation.cc:407] Connection header received
0929 15:51:49.811489 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:49.811495 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:49.811690 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:49.811835 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:49.813095 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:49.814191 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:49.815379 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:49.816148 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:49.816247 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:49.819928 (+ 3681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:49.819938 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:49.819950 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:49.819984 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:49.822650 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:49.823224 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:49.823231 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:49.823233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:49.823352 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:49.823659 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:49.823662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:49.823664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:49.823989 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:49.824199 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:49.824403 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 15:51:49.824577 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":348,"threads_started":1}
Sep 29 15:51:49 dist-test-slave-r0ck krb5kdc[3960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:51:49.841503 4196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:49.829240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59582 (local address 127.0.97.62:33851)
0929 15:51:49.829480 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:49.829486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:49.829677 (+ 191us) server_negotiation.cc:407] Connection header received
0929 15:51:49.829869 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:49.829876 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:49.829949 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:49.830048 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:49.831166 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:49.831858 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:49.833031 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:49.833501 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:49.833626 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:49.837175 (+ 3549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:49.837180 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:49.837183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:49.837212 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:49.839479 (+ 2267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:49.839998 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:49.840001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:49.840003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:49.840103 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:49.840376 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:49.840379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:49.840381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:49.840649 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:49.840820 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:49.841036 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 15:51:49.841234 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260929 15:51:49.842033 4005 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59582
I20260929 15:51:49.842972 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 4066
I20260929 15:51:49.854153 4468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:49.843733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32992 (local address 127.0.97.1:34265)
0929 15:51:49.843949 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:49.843955 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:49.844118 (+ 163us) server_negotiation.cc:407] Connection header received
0929 15:51:49.844216 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:49.844219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:49.844301 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:49.844372 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:49.845563 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:49.846190 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:49.847276 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:49.847673 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:49.847762 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:49.849908 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:49.849913 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:49.849916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:49.849943 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:49.852077 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:49.852595 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:49.852598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:49.852600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:49.852702 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:49.852988 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:49.852991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:49.852993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:49.853201 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:49.853442 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:49.853669 (+ 227us) server_negotiation.cc:299] Negotiation successful
0929 15:51:49.853830 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 15:51:49.854671 4189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:51:49.935542 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 4066
I20260929 15:51:49.951370 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 4200
Sep 29 15:51:49 dist-test-slave-r0ck krb5kdc[3960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 15:51:49.966588 4473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:49.952204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38170 (local address 127.0.97.2:33111)
0929 15:51:49.952768 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:49.952775 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:49.952806 (+ 31us) server_negotiation.cc:407] Connection header received
0929 15:51:49.952864 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:49.952870 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:49.953073 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:49.953201 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:49.954345 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:49.955353 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:49.956551 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:49.957082 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:49.957179 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:49.961240 (+ 4061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:49.961252 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:49.961266 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:49.961309 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:49.964092 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:49.964688 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:49.964695 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:49.964697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:49.964814 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:49.965113 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:49.965116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:49.965118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:49.965503 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:49.965715 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:49.965971 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 15:51:49.966161 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":322,"threads_started":1}
I20260929 15:51:49.967891 4323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:51:50.050017 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 4200
I20260929 15:51:50.065914 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 4333
Sep 29 15:51:50 dist-test-slave-r0ck krb5kdc[3960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 15:51:50.081198 4475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:50.066838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33062 (local address 127.0.97.3:46681)
0929 15:51:50.067503 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:50.067510 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:50.067539 (+ 29us) server_negotiation.cc:407] Connection header received
0929 15:51:50.067600 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:50.067606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:50.067805 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:50.067948 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:50.069141 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:50.070221 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:50.071455 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:50.071974 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:50.072068 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:50.076136 (+ 4068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:50.076146 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:50.076160 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:50.076193 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:50.078893 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:50.079459 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:50.079466 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:50.079468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:50.079582 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:50.079892 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:50.079895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:50.079897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:50.080216 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:50.080444 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:50.080620 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 15:51:50.080797 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":371,"threads_started":1}
I20260929 15:51:50.082585 4456 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:51:50.164428 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 4333
I20260929 15:51:50.179690 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 3973
I20260929 15:51:50.180998 4035 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:51:50.282097 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 3973
2026-09-29T15:51:50Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3781 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:51:50 dist-test-slave-r0ck krb5kdc[4480](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:51:50 dist-test-slave-r0ck krb5kdc[4480](info): set up 2 sockets
Sep 29 15:51:50 dist-test-slave-r0ck krb5kdc[4480](info): commencing operation
krb5kdc: starting...
W20260929 15:51:51.369817 388 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.005s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:51:51 dist-test-slave-r0ck krb5kdc[4480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:51:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:51:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:51:51.559183 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43369
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:51:51.872189 4493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:51:51.872474 4493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:51:51.872565 4493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:51:51.879773 4493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:51:51.879899 4493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:51:51.879945 4493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:51:51.879998 4493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:51:51.880048 4493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:51:51.889775 4493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:51:51.890965 4493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:51:51.892185 4493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:51:51.903122 4498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:51.903172 4499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:51.903203 4501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:51:51.904241 4493 server_base.cc:1034] running on GCE node
I20260929 15:51:51.904947 4493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:51:51.906253 4493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:51:51.907495 4493 hybrid_clock.cc:648] HybridClock initialized: now 1790697111907453 us; error 59 us; skew 500 ppm
I20260929 15:51:51.907845 4493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:51:51 dist-test-slave-r0ck krb5kdc[4480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:51:51.910815 4493 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:51:51.911891 4493 fs_manager.cc:359] Metadata directory not provided
I20260929 15:51:51.912022 4493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:51:51.912323 4493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:51:51.915162 4493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "5107030e1ffb44459f410a880475f86e"
format_stamp: "Formatted at 2026-09-29 15:51:51 on dist-test-slave-r0ck"
server_key: "e1ad8e7c8036a9c7272e0a1c701c30f0"
server_key_iv: "e262397fbeb35704043061cf08b16906"
server_key_version: "encryptionkey@0"
I20260929 15:51:51.915948 4493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "5107030e1ffb44459f410a880475f86e"
format_stamp: "Formatted at 2026-09-29 15:51:51 on dist-test-slave-r0ck"
server_key: "e1ad8e7c8036a9c7272e0a1c701c30f0"
server_key_iv: "e262397fbeb35704043061cf08b16906"
server_key_version: "encryptionkey@0"
I20260929 15:51:51.920840 4493 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 15:51:51.924290 4507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:51.925673 4493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:51:51.925850 4493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "5107030e1ffb44459f410a880475f86e"
format_stamp: "Formatted at 2026-09-29 15:51:51 on dist-test-slave-r0ck"
server_key: "e1ad8e7c8036a9c7272e0a1c701c30f0"
server_key_iv: "e262397fbeb35704043061cf08b16906"
server_key_version: "encryptionkey@0"
I20260929 15:51:51.926059 4493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:51:51.936611 4493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:51:51.940225 4493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:51:51.940501 4493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:51:51.958350 4493 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43369
I20260929 15:51:51.958369 4558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43369 every 8 connection(s)
I20260929 15:51:51.959194 4493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:51:51.962666 4559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:51.966099 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 4493
I20260929 15:51:51.966332 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:51:51.966933 388 external_mini_cluster.cc:1428] Setting key cb87a456aa1c83ed0d0420365a361ada
I20260929 15:51:51.975775 4559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5107030e1ffb44459f410a880475f86e: Bootstrap starting.
I20260929 15:51:51.979344 4559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5107030e1ffb44459f410a880475f86e: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:51.980329 4559 log.cc:824] T 00000000000000000000000000000000 P 5107030e1ffb44459f410a880475f86e: Log is configured to *not* fsync() on all Append() calls
Sep 29 15:51:51 dist-test-slave-r0ck krb5kdc[4480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:51:51.983621 4559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5107030e1ffb44459f410a880475f86e: No bootstrap required, opened a new log
I20260929 15:51:51.990549 4562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:51.968930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43820 (local address 127.0.97.62:43369)
0929 15:51:51.969806 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:51.969822 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:51.969871 (+ 49us) server_negotiation.cc:407] Connection header received
0929 15:51:51.971546 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:51.971579 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:51.972085 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:51.972533 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:51.973903 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:51.975816 (+ 1913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:51.977146 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:51.978186 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:51.978361 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:51.982407 (+ 4046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:51.982416 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:51.982435 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:51.982490 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:51.986259 (+ 3769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:51.986897 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:51.986911 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:51.986919 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:51.987120 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:51.987443 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:51.987448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:51.987451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:51.988111 (+ 660us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:51.988370 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:51.988756 (+ 386us) server_negotiation.cc:299] Negotiation successful
0929 15:51:51.989192 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":400,"threads_started":1}
I20260929 15:51:51.993789 4559 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5107030e1ffb44459f410a880475f86e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5107030e1ffb44459f410a880475f86e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43369 } }
I20260929 15:51:51.994104 4559 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5107030e1ffb44459f410a880475f86e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:51.994174 4559 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5107030e1ffb44459f410a880475f86e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5107030e1ffb44459f410a880475f86e, State: Initialized, Role: FOLLOWER
I20260929 15:51:51.994607 4559 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5107030e1ffb44459f410a880475f86e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5107030e1ffb44459f410a880475f86e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43369 } }
I20260929 15:51:51.994787 4559 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5107030e1ffb44459f410a880475f86e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:51:51.994879 4559 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5107030e1ffb44459f410a880475f86e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:51:51.995009 4559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5107030e1ffb44459f410a880475f86e [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:51.996989 4559 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5107030e1ffb44459f410a880475f86e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5107030e1ffb44459f410a880475f86e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43369 } }
I20260929 15:51:51.997478 4559 leader_election.cc:302] T 00000000000000000000000000000000 P 5107030e1ffb44459f410a880475f86e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5107030e1ffb44459f410a880475f86e; no voters:
I20260929 15:51:51.997998 4559 leader_election.cc:288] T 00000000000000000000000000000000 P 5107030e1ffb44459f410a880475f86e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:51:51.998143 4564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5107030e1ffb44459f410a880475f86e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:51:51.999106 4564 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5107030e1ffb44459f410a880475f86e [term 1 LEADER]: Becoming Leader. State: Replica: 5107030e1ffb44459f410a880475f86e, State: Running, Role: LEADER
I20260929 15:51:51.999644 4564 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5107030e1ffb44459f410a880475f86e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5107030e1ffb44459f410a880475f86e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43369 } }
I20260929 15:51:52.000167 4559 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5107030e1ffb44459f410a880475f86e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:51:52.002089 4566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5107030e1ffb44459f410a880475f86e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5107030e1ffb44459f410a880475f86e. Latest consensus state: current_term: 1 leader_uuid: "5107030e1ffb44459f410a880475f86e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5107030e1ffb44459f410a880475f86e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43369 } } }
I20260929 15:51:52.002439 4566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5107030e1ffb44459f410a880475f86e [sys.catalog]: This master's current role is: LEADER
I20260929 15:51:52.003012 4565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5107030e1ffb44459f410a880475f86e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5107030e1ffb44459f410a880475f86e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5107030e1ffb44459f410a880475f86e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43369 } } }
I20260929 15:51:52.003413 4565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5107030e1ffb44459f410a880475f86e [sys.catalog]: This master's current role is: LEADER
I20260929 15:51:52.004537 4570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:51:52.009255 4570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:51:52.018019 4570 catalog_manager.cc:1347] Generated new cluster ID: 30e429013cec4f2286a9a2acb4852337
I20260929 15:51:52.018155 4570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:51:52.028644 4570 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:51:52.030556 4570 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:51:52.040189 4570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5107030e1ffb44459f410a880475f86e: Generated new TSK 0
I20260929 15:51:52.041208 4570 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 15:51:52.088603 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43369
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:51:52.401547 4585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:51:52.402050 4585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:51:52.402259 4585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:51:52.409739 4585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:51:52.409835 4585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:51:52.409957 4585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:51:52.419394 4585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:51:52.420562 4585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:51:52.421845 4585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:51:52.434267 4590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:52.434307 4591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:52.434228 4593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:51:52.438931 4585 server_base.cc:1034] running on GCE node
I20260929 15:51:52.439563 4585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:51:52.440544 4585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:51:52.441804 4585 hybrid_clock.cc:648] HybridClock initialized: now 1790697112441716 us; error 54 us; skew 500 ppm
I20260929 15:51:52.442131 4585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:51:52 dist-test-slave-r0ck krb5kdc[4480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:51:52.445446 4585 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:51:52.446741 4585 fs_manager.cc:359] Metadata directory not provided
I20260929 15:51:52.446878 4585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:51:52.447180 4585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:51:52.450137 4585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-0/data/instance:
uuid: "e4dc73fed3fc4840abdc1670d2ba0e90"
format_stamp: "Formatted at 2026-09-29 15:51:52 on dist-test-slave-r0ck"
server_key: "980237134a7b13f3dcdf0635a665d695"
server_key_iv: "b96a6c28eae0424da70d4e629a52ba82"
server_key_version: "encryptionkey@0"
I20260929 15:51:52.450984 4585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e4dc73fed3fc4840abdc1670d2ba0e90"
format_stamp: "Formatted at 2026-09-29 15:51:52 on dist-test-slave-r0ck"
server_key: "980237134a7b13f3dcdf0635a665d695"
server_key_iv: "b96a6c28eae0424da70d4e629a52ba82"
server_key_version: "encryptionkey@0"
I20260929 15:51:52.456099 4585 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 15:51:52.459484 4599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:52.460945 4585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 15:51:52.461126 4585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-0/wal
uuid: "e4dc73fed3fc4840abdc1670d2ba0e90"
format_stamp: "Formatted at 2026-09-29 15:51:52 on dist-test-slave-r0ck"
server_key: "980237134a7b13f3dcdf0635a665d695"
server_key_iv: "b96a6c28eae0424da70d4e629a52ba82"
server_key_version: "encryptionkey@0"
I20260929 15:51:52.461331 4585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:51:52.482451 4585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:51:52.486835 4585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:51:52.487104 4585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:51:52.488001 4585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:51:52.489262 4585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:51:52.489396 4585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:52.489521 4585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:51:52.489619 4585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:52.520788 4585 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45985
I20260929 15:51:52.520888 4711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45985 every 8 connection(s)
I20260929 15:51:52.521692 4585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:51:52.527281 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 4585
I20260929 15:51:52.527515 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:51:52.528110 388 external_mini_cluster.cc:1428] Setting key b2281d39605139d9f6f52c1f8c4ffcbf
Sep 29 15:51:52 dist-test-slave-r0ck krb5kdc[4480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:51:52.541633 4714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:52.524070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49609 (local address 127.0.97.62:43369)
0929 15:51:52.524484 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:52.524490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:52.525522 (+ 1032us) server_negotiation.cc:407] Connection header received
0929 15:51:52.527241 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:52.527246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:52.527334 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:52.527462 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:52.528996 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:52.530022 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:52.531621 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:52.532341 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:52.532490 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:52.535878 (+ 3388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:52.535883 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:52.535886 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:52.535916 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:52.538442 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:52.539128 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:52.539135 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:52.539139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:52.539288 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:52.539777 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:52.539782 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:52.539785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:52.540202 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:52.540358 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:52.541151 (+ 793us) server_negotiation.cc:299] Negotiation successful
0929 15:51:52.541320 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":192,"threads_started":1}
I20260929 15:51:52.543116 4715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:52.524651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43369 (local address 127.0.97.1:49609)
0929 15:51:52.525310 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:52.525406 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:52.526988 (+ 1582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:52.527723 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:52.527731 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:52.528310 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:51:52.528843 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:52.528861 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:52.530159 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:52.530162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:52.531473 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:52.531485 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:52.532530 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:52.532535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:52.532701 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:52.533423 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:51:52.533444 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:51:52.535661 (+ 2217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:51:52.538579 (+ 2918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:52.538588 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:52.538602 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:52.539009 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:52.539418 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:52.539425 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:52.539431 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:52.539649 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:52.540339 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:51:52.540350 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:51:52.540772 (+ 422us) client_negotiation.cc:769] Sending connection context
0929 15:51:52.541083 (+ 311us) client_negotiation.cc:240] Negotiation successful
0929 15:51:52.541663 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":280,"threads_started":1}
I20260929 15:51:52.544641 4712 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43369
I20260929 15:51:52.544982 4712 heartbeater.cc:464] Registering TS with master...
I20260929 15:51:52.545763 4712 heartbeater.cc:511] Master 127.0.97.62:43369 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 15:51:52.547891 4524 ts_manager.cc:194] Registered new tserver with Master: e4dc73fed3fc4840abdc1670d2ba0e90 (127.0.97.1:45985)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 15:51:52.549504 4524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49609
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 15:51:52.570786 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43369
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:51:52.885468 4718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:51:52.885802 4718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:51:52.885983 4718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:51:52.893198 4718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:51:52.893349 4718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:51:52.893549 4718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:51:52.903390 4718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:51:52.904744 4718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:51:52.906122 4718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:51:52.918545 4723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:52.918579 4724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:52.918645 4726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:51:52.922794 4718 server_base.cc:1034] running on GCE node
I20260929 15:51:52.923445 4718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:51:52.924400 4718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:51:52.925626 4718 hybrid_clock.cc:648] HybridClock initialized: now 1790697112925575 us; error 34 us; skew 500 ppm
I20260929 15:51:52.925972 4718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:51:52 dist-test-slave-r0ck krb5kdc[4480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:51:52.928927 4718 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:51:52.930310 4718 fs_manager.cc:359] Metadata directory not provided
I20260929 15:51:52.930451 4718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:51:52.930734 4718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:51:52.933538 4718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-1/data/instance:
uuid: "3b28e6c9e0414b90b1fb162ad0b440d7"
format_stamp: "Formatted at 2026-09-29 15:51:52 on dist-test-slave-r0ck"
server_key: "af592078990c30381e25a2ea95c1317d"
server_key_iv: "1418d991edd24ce595fd5cc69ec9f204"
server_key_version: "encryptionkey@0"
I20260929 15:51:52.934265 4718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "3b28e6c9e0414b90b1fb162ad0b440d7"
format_stamp: "Formatted at 2026-09-29 15:51:52 on dist-test-slave-r0ck"
server_key: "af592078990c30381e25a2ea95c1317d"
server_key_iv: "1418d991edd24ce595fd5cc69ec9f204"
server_key_version: "encryptionkey@0"
I20260929 15:51:52.938972 4718 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 15:51:52.942170 4732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:52.944337 4718 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 15:51:52.944527 4718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-1/wal
uuid: "3b28e6c9e0414b90b1fb162ad0b440d7"
format_stamp: "Formatted at 2026-09-29 15:51:52 on dist-test-slave-r0ck"
server_key: "af592078990c30381e25a2ea95c1317d"
server_key_iv: "1418d991edd24ce595fd5cc69ec9f204"
server_key_version: "encryptionkey@0"
I20260929 15:51:52.944742 4718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:51:52.969635 4718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:51:52.974820 4718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:51:52.975049 4718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:51:52.975922 4718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:51:52.977195 4718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:51:52.977280 4718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:52.977412 4718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:51:52.977474 4718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:53.008878 4718 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42051
I20260929 15:51:53.008985 4844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42051 every 8 connection(s)
I20260929 15:51:53.009740 4718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:51:53.017858 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 4718
I20260929 15:51:53.018059 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:51:53.018566 388 external_mini_cluster.cc:1428] Setting key 85730a52b3261a12340f88c0bfeb1b57
Sep 29 15:51:53 dist-test-slave-r0ck krb5kdc[4480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:51:53.029953 4714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:53.012297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38329 (local address 127.0.97.62:43369)
0929 15:51:53.012507 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:53.012513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:53.013705 (+ 1192us) server_negotiation.cc:407] Connection header received
0929 15:51:53.015493 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:53.015498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:53.015566 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:53.015659 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:53.017177 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:53.018298 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:53.019924 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:53.020342 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:53.020479 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:53.023885 (+ 3406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:53.023890 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:53.023894 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:53.023923 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:53.026579 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:53.027492 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:53.027496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:53.027498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:53.027620 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:53.028121 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:53.028125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:53.028127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:53.028381 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:53.028594 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:53.029460 (+ 866us) server_negotiation.cc:299] Negotiation successful
0929 15:51:53.029689 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 15:51:53.031376 4847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:53.012773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43369 (local address 127.0.97.2:38329)
0929 15:51:53.013556 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:53.013594 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:53.015224 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:53.015862 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:53.015869 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:53.016391 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:51:53.016925 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:53.016941 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:53.018452 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:53.018456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:53.019767 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:53.019779 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:53.020558 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:53.020565 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:53.020776 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:53.021546 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:51:53.021569 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:51:53.023660 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:51:53.026726 (+ 3066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:53.026737 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:53.026757 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:53.027377 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:53.027732 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:53.027739 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:53.027746 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:53.027984 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:53.028570 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:51:53.028581 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:51:53.029033 (+ 452us) client_negotiation.cc:769] Sending connection context
0929 15:51:53.029327 (+ 294us) client_negotiation.cc:240] Negotiation successful
0929 15:51:53.029933 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":307,"threads_started":1}
I20260929 15:51:53.032909 4845 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43369
I20260929 15:51:53.033215 4845 heartbeater.cc:464] Registering TS with master...
I20260929 15:51:53.033991 4845 heartbeater.cc:511] Master 127.0.97.62:43369 requested a full tablet report, sending...
I20260929 15:51:53.035468 4524 ts_manager.cc:194] Registered new tserver with Master: 3b28e6c9e0414b90b1fb162ad0b440d7 (127.0.97.2:42051)
I20260929 15:51:53.036247 4524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38329
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 15:51:53.064929 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43369
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:51:53.375953 4850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:51:53.376307 4850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:51:53.376503 4850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:51:53.383772 4850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:51:53.383929 4850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:51:53.384119 4850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:51:53.393971 4850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:51:53.395308 4850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:51:53.396589 4850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:51:53.410252 4858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:53.410354 4855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:53.410322 4856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:51:53.410916 4850 server_base.cc:1034] running on GCE node
I20260929 15:51:53.411546 4850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:51:53.412422 4850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:51:53.413686 4850 hybrid_clock.cc:648] HybridClock initialized: now 1790697113413648 us; error 33 us; skew 500 ppm
I20260929 15:51:53.414039 4850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:51:53 dist-test-slave-r0ck krb5kdc[4480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:51:53.417143 4850 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:51:53.418315 4850 fs_manager.cc:359] Metadata directory not provided
I20260929 15:51:53.418403 4850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:51:53.418653 4850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:51:53.421649 4850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-2/data/instance:
uuid: "fd1a77d0d9474f29840491fcb6d1e37c"
format_stamp: "Formatted at 2026-09-29 15:51:53 on dist-test-slave-r0ck"
server_key: "b0a4abdb9d0964c1ca21f1f91ca583cc"
server_key_iv: "de82c8f7d3f55733a0472a3f3b3d8a65"
server_key_version: "encryptionkey@0"
I20260929 15:51:53.422448 4850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "fd1a77d0d9474f29840491fcb6d1e37c"
format_stamp: "Formatted at 2026-09-29 15:51:53 on dist-test-slave-r0ck"
server_key: "b0a4abdb9d0964c1ca21f1f91ca583cc"
server_key_iv: "de82c8f7d3f55733a0472a3f3b3d8a65"
server_key_version: "encryptionkey@0"
I20260929 15:51:53.427436 4850 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 15:51:53.430930 4864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:53.432262 4850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 15:51:53.432399 4850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-2/wal
uuid: "fd1a77d0d9474f29840491fcb6d1e37c"
format_stamp: "Formatted at 2026-09-29 15:51:53 on dist-test-slave-r0ck"
server_key: "b0a4abdb9d0964c1ca21f1f91ca583cc"
server_key_iv: "de82c8f7d3f55733a0472a3f3b3d8a65"
server_key_version: "encryptionkey@0"
I20260929 15:51:53.432575 4850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:51:53.450017 4850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:51:53.454260 4850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:51:53.454476 4850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:51:53.455312 4850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:51:53.456730 4850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:51:53.456825 4850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:53.456898 4850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:51:53.456944 4850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:53.489784 4850 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38021
I20260929 15:51:53.489816 4976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38021 every 8 connection(s)
I20260929 15:51:53.490594 4850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:51:53.492635 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 4850
I20260929 15:51:53.492895 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:51:53.493399 388 external_mini_cluster.cc:1428] Setting key 9a8e81f1b7234eebe00bdbd3368fa9e6
Sep 29 15:51:53 dist-test-slave-r0ck krb5kdc[4480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:51:53.510079 4714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:53.493030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55861 (local address 127.0.97.62:43369)
0929 15:51:53.493234 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:53.493240 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:53.494546 (+ 1306us) server_negotiation.cc:407] Connection header received
0929 15:51:53.496334 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:53.496338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:53.496438 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:53.496544 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:53.498024 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:53.498745 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:53.500304 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:53.500786 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:53.500916 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:53.504278 (+ 3362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:53.504283 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:53.504286 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:53.504332 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:53.506960 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:53.507866 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:53.507871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:53.507876 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:53.508015 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:53.508535 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:53.508540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:53.508543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:53.508818 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:53.508938 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:53.509680 (+ 742us) server_negotiation.cc:299] Negotiation successful
0929 15:51:53.509838 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 15:51:53.511025 4979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:53.493597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43369 (local address 127.0.97.3:55861)
0929 15:51:53.494397 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:53.494439 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:53.496087 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:53.496767 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:53.496775 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:53.497274 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:51:53.497857 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:53.497877 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:53.498904 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:53.498908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:53.500168 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:53.500179 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:53.501017 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:53.501025 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:53.501258 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:53.501952 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:51:53.501975 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:51:53.504057 (+ 2082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:51:53.507090 (+ 3033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:53.507099 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:53.507115 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:53.507747 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:53.508125 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:51:53.508130 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:51:53.508133 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:51:53.508401 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:51:53.508930 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:51:53.508935 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:51:53.509262 (+ 327us) client_negotiation.cc:769] Sending connection context
0929 15:51:53.509565 (+ 303us) client_negotiation.cc:240] Negotiation successful
0929 15:51:53.509959 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":372,"threads_started":1}
I20260929 15:51:53.512445 4977 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43369
I20260929 15:51:53.512712 4977 heartbeater.cc:464] Registering TS with master...
I20260929 15:51:53.513324 4977 heartbeater.cc:511] Master 127.0.97.62:43369 requested a full tablet report, sending...
I20260929 15:51:53.514698 4524 ts_manager.cc:194] Registered new tserver with Master: fd1a77d0d9474f29840491fcb6d1e37c (127.0.97.3:38021)
I20260929 15:51:53.515517 4524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55861
I20260929 15:51:53.520459 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:51:53.536742 4714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:53.525091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43830 (local address 127.0.97.62:43369)
0929 15:51:53.525309 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:53.525315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:53.525646 (+ 331us) server_negotiation.cc:407] Connection header received
0929 15:51:53.525762 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:53.525765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:53.525818 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:53.525918 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:53.527263 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:53.527928 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:53.529099 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:53.529579 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:53.529673 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:53.531845 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:53.531850 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:53.531853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:53.531900 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:53.534231 (+ 2331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:53.534857 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:53.534864 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:53.534872 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:53.534984 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:53.535373 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:53.535379 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:53.535382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:53.535662 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:53.535922 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:53.536219 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 15:51:53.536456 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 15:51:53.540483 4524 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:51:53.547045 4524 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:51:53.552402 4712 heartbeater.cc:503] Master 127.0.97.62:43369 was elected leader, sending a full tablet report...
I20260929 15:51:53.580228 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:53.558726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60218 (local address 127.0.97.3:38021)
0929 15:51:53.573408 (+ 14682us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:53.573418 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:53.573452 (+ 34us) server_negotiation.cc:407] Connection header received
0929 15:51:53.573550 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:53.573559 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:53.573756 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:53.573933 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:51:53.574412 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:53.575811 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:53.577808 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:53.579133 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:53.579276 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:53.579517 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:53.579742 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 15:51:53.579892 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14488,"thread_start_us":14324,"threads_started":1}
I20260929 15:51:53.580387 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:53.558481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38021 (local address 127.0.0.1:60218)
0929 15:51:53.570829 (+ 12348us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:53.570874 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:53.571067 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:53.573933 (+ 2866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:53.573937 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:53.573959 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:51:53.574257 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:53.574265 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:53.575951 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:53.575958 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:53.577668 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:53.577683 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:53.579309 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:53.579314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:53.579500 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:53.579553 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 15:51:53.579705 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 15:51:53.579890 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12049,"spinlock_wait_cycles":5376,"thread_start_us":12725,"threads_started":1}
I20260929 15:51:53.581252 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:53.558534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45985 (local address 127.0.0.1:42692)
0929 15:51:53.570873 (+ 12339us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:53.570889 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:53.571067 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:53.572951 (+ 1884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:53.572958 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:53.573003 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:51:53.573564 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:53.573575 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:53.575553 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:53.575558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:53.578292 (+ 2734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:53.578306 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:53.580393 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:53.580398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:53.580623 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:53.580778 (+ 155us) client_negotiation.cc:769] Sending connection context
0929 15:51:53.580852 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 15:51:53.580949 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12232,"mutex_wait_us":389,"spinlock_wait_cycles":11648,"thread_start_us":11846,"threads_started":1}
I20260929 15:51:53.581329 4992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:53.558989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42051 (local address 127.0.0.1:35288)
0929 15:51:53.571976 (+ 12987us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:53.571994 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:53.572109 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:53.572936 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:53.572946 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:53.572995 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:51:53.573670 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:53.573682 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:53.575518 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:53.575523 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:53.578292 (+ 2769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:53.578308 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:53.580352 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:53.580360 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:53.580623 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:53.580678 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 15:51:53.580757 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 15:51:53.580879 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12851,"mutex_wait_us":268,"spinlock_wait_cycles":5376,"thread_start_us":12776,"threads_started":1}
I20260929 15:51:53.581938 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:53.559553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42692 (local address 127.0.97.1:45985)
0929 15:51:53.571957 (+ 12404us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:53.571969 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:53.572006 (+ 37us) server_negotiation.cc:407] Connection header received
0929 15:51:53.572124 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:53.572134 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:53.572440 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:53.572661 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:51:53.573709 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:53.575357 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:53.578468 (+ 3111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:53.579852 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:53.579962 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:53.580165 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:53.581564 (+ 1399us) server_negotiation.cc:299] Negotiation successful
0929 15:51:53.581708 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12207,"thread_start_us":12008,"threads_started":1}
I20260929 15:51:53.582064 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:53.560026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35288 (local address 127.0.97.2:42051)
0929 15:51:53.571539 (+ 11513us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:53.571551 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:53.572224 (+ 673us) server_negotiation.cc:407] Connection header received
0929 15:51:53.572316 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:53.572323 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:53.572556 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:53.572726 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:51:53.573855 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:53.575357 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:53.578468 (+ 3111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:53.579843 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:53.579962 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:53.580133 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:53.581592 (+ 1459us) server_negotiation.cc:299] Negotiation successful
0929 15:51:53.581766 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11350,"thread_start_us":11144,"threads_started":1}
I20260929 15:51:53.584556 4912 tablet_service.cc:1467] Processing CreateTablet for tablet 5d37624a8979474c923ad507aa8e8573 (DEFAULT_TABLE table=test-table [id=04e65b30aca84cca8f7009c47d808fe8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:51:53.585103 4780 tablet_service.cc:1467] Processing CreateTablet for tablet 5d37624a8979474c923ad507aa8e8573 (DEFAULT_TABLE table=test-table [id=04e65b30aca84cca8f7009c47d808fe8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:51:53.585493 4647 tablet_service.cc:1467] Processing CreateTablet for tablet 5d37624a8979474c923ad507aa8e8573 (DEFAULT_TABLE table=test-table [id=04e65b30aca84cca8f7009c47d808fe8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:51:53.586112 4912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d37624a8979474c923ad507aa8e8573. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:53.586297 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d37624a8979474c923ad507aa8e8573. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:53.586692 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d37624a8979474c923ad507aa8e8573. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:53.596817 4995 tablet_bootstrap.cc:492] T 5d37624a8979474c923ad507aa8e8573 P 3b28e6c9e0414b90b1fb162ad0b440d7: Bootstrap starting.
I20260929 15:51:53.599511 4995 tablet_bootstrap.cc:654] T 5d37624a8979474c923ad507aa8e8573 P 3b28e6c9e0414b90b1fb162ad0b440d7: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:53.600368 4996 tablet_bootstrap.cc:492] T 5d37624a8979474c923ad507aa8e8573 P fd1a77d0d9474f29840491fcb6d1e37c: Bootstrap starting.
I20260929 15:51:53.601179 4995 log.cc:824] T 5d37624a8979474c923ad507aa8e8573 P 3b28e6c9e0414b90b1fb162ad0b440d7: Log is configured to *not* fsync() on all Append() calls
I20260929 15:51:53.601938 4997 tablet_bootstrap.cc:492] T 5d37624a8979474c923ad507aa8e8573 P e4dc73fed3fc4840abdc1670d2ba0e90: Bootstrap starting.
I20260929 15:51:53.604553 4996 tablet_bootstrap.cc:654] T 5d37624a8979474c923ad507aa8e8573 P fd1a77d0d9474f29840491fcb6d1e37c: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:53.604975 4997 tablet_bootstrap.cc:654] T 5d37624a8979474c923ad507aa8e8573 P e4dc73fed3fc4840abdc1670d2ba0e90: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:53.606001 4997 log.cc:824] T 5d37624a8979474c923ad507aa8e8573 P e4dc73fed3fc4840abdc1670d2ba0e90: Log is configured to *not* fsync() on all Append() calls
I20260929 15:51:53.606053 4996 log.cc:824] T 5d37624a8979474c923ad507aa8e8573 P fd1a77d0d9474f29840491fcb6d1e37c: Log is configured to *not* fsync() on all Append() calls
I20260929 15:51:53.609427 4997 tablet_bootstrap.cc:492] T 5d37624a8979474c923ad507aa8e8573 P e4dc73fed3fc4840abdc1670d2ba0e90: No bootstrap required, opened a new log
I20260929 15:51:53.609442 4996 tablet_bootstrap.cc:492] T 5d37624a8979474c923ad507aa8e8573 P fd1a77d0d9474f29840491fcb6d1e37c: No bootstrap required, opened a new log
I20260929 15:51:53.609441 4995 tablet_bootstrap.cc:492] T 5d37624a8979474c923ad507aa8e8573 P 3b28e6c9e0414b90b1fb162ad0b440d7: No bootstrap required, opened a new log
I20260929 15:51:53.609704 4996 ts_tablet_manager.cc:1397] T 5d37624a8979474c923ad507aa8e8573 P fd1a77d0d9474f29840491fcb6d1e37c: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260929 15:51:53.609704 4995 ts_tablet_manager.cc:1397] T 5d37624a8979474c923ad507aa8e8573 P 3b28e6c9e0414b90b1fb162ad0b440d7: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.003s
I20260929 15:51:53.609771 4997 ts_tablet_manager.cc:1397] T 5d37624a8979474c923ad507aa8e8573 P e4dc73fed3fc4840abdc1670d2ba0e90: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20260929 15:51:53.616425 4997 raft_consensus.cc:348] T 5d37624a8979474c923ad507aa8e8573 P e4dc73fed3fc4840abdc1670d2ba0e90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1a77d0d9474f29840491fcb6d1e37c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38021 } } peers { permanent_uuid: "e4dc73fed3fc4840abdc1670d2ba0e90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45985 } } peers { permanent_uuid: "3b28e6c9e0414b90b1fb162ad0b440d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42051 } }
I20260929 15:51:53.616775 4997 raft_consensus.cc:374] T 5d37624a8979474c923ad507aa8e8573 P e4dc73fed3fc4840abdc1670d2ba0e90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:53.616882 4997 raft_consensus.cc:729] T 5d37624a8979474c923ad507aa8e8573 P e4dc73fed3fc4840abdc1670d2ba0e90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4dc73fed3fc4840abdc1670d2ba0e90, State: Initialized, Role: FOLLOWER
I20260929 15:51:53.617542 4997 consensus_queue.cc:260] T 5d37624a8979474c923ad507aa8e8573 P e4dc73fed3fc4840abdc1670d2ba0e90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1a77d0d9474f29840491fcb6d1e37c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38021 } } peers { permanent_uuid: "e4dc73fed3fc4840abdc1670d2ba0e90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45985 } } peers { permanent_uuid: "3b28e6c9e0414b90b1fb162ad0b440d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42051 } }
I20260929 15:51:53.619235 4997 ts_tablet_manager.cc:1428] T 5d37624a8979474c923ad507aa8e8573 P e4dc73fed3fc4840abdc1670d2ba0e90: Time spent starting tablet: real 0.009s user 0.008s sys 0.001s
I20260929 15:51:53.620136 4996 raft_consensus.cc:348] T 5d37624a8979474c923ad507aa8e8573 P fd1a77d0d9474f29840491fcb6d1e37c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1a77d0d9474f29840491fcb6d1e37c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38021 } } peers { permanent_uuid: "e4dc73fed3fc4840abdc1670d2ba0e90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45985 } } peers { permanent_uuid: "3b28e6c9e0414b90b1fb162ad0b440d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42051 } }
I20260929 15:51:53.620136 4995 raft_consensus.cc:348] T 5d37624a8979474c923ad507aa8e8573 P 3b28e6c9e0414b90b1fb162ad0b440d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1a77d0d9474f29840491fcb6d1e37c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38021 } } peers { permanent_uuid: "e4dc73fed3fc4840abdc1670d2ba0e90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45985 } } peers { permanent_uuid: "3b28e6c9e0414b90b1fb162ad0b440d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42051 } }
I20260929 15:51:53.620532 4995 raft_consensus.cc:374] T 5d37624a8979474c923ad507aa8e8573 P 3b28e6c9e0414b90b1fb162ad0b440d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:53.620560 4996 raft_consensus.cc:374] T 5d37624a8979474c923ad507aa8e8573 P fd1a77d0d9474f29840491fcb6d1e37c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:53.620651 4995 raft_consensus.cc:729] T 5d37624a8979474c923ad507aa8e8573 P 3b28e6c9e0414b90b1fb162ad0b440d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b28e6c9e0414b90b1fb162ad0b440d7, State: Initialized, Role: FOLLOWER
I20260929 15:51:53.620669 4996 raft_consensus.cc:729] T 5d37624a8979474c923ad507aa8e8573 P fd1a77d0d9474f29840491fcb6d1e37c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd1a77d0d9474f29840491fcb6d1e37c, State: Initialized, Role: FOLLOWER
I20260929 15:51:53.621263 4996 consensus_queue.cc:260] T 5d37624a8979474c923ad507aa8e8573 P fd1a77d0d9474f29840491fcb6d1e37c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1a77d0d9474f29840491fcb6d1e37c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38021 } } peers { permanent_uuid: "e4dc73fed3fc4840abdc1670d2ba0e90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45985 } } peers { permanent_uuid: "3b28e6c9e0414b90b1fb162ad0b440d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42051 } }
I20260929 15:51:53.621238 4995 consensus_queue.cc:260] T 5d37624a8979474c923ad507aa8e8573 P 3b28e6c9e0414b90b1fb162ad0b440d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1a77d0d9474f29840491fcb6d1e37c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38021 } } peers { permanent_uuid: "e4dc73fed3fc4840abdc1670d2ba0e90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45985 } } peers { permanent_uuid: "3b28e6c9e0414b90b1fb162ad0b440d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42051 } }
I20260929 15:51:53.622423 4845 heartbeater.cc:503] Master 127.0.97.62:43369 was elected leader, sending a full tablet report...
I20260929 15:51:53.623304 4995 ts_tablet_manager.cc:1428] T 5d37624a8979474c923ad507aa8e8573 P 3b28e6c9e0414b90b1fb162ad0b440d7: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260929 15:51:53.624758 4977 heartbeater.cc:503] Master 127.0.97.62:43369 was elected leader, sending a full tablet report...
I20260929 15:51:53.625376 4996 ts_tablet_manager.cc:1428] T 5d37624a8979474c923ad507aa8e8573 P fd1a77d0d9474f29840491fcb6d1e37c: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260929 15:51:53.653769 5003 raft_consensus.cc:482] T 5d37624a8979474c923ad507aa8e8573 P fd1a77d0d9474f29840491fcb6d1e37c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:51:53.654021 5003 raft_consensus.cc:504] T 5d37624a8979474c923ad507aa8e8573 P fd1a77d0d9474f29840491fcb6d1e37c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1a77d0d9474f29840491fcb6d1e37c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38021 } } peers { permanent_uuid: "e4dc73fed3fc4840abdc1670d2ba0e90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45985 } } peers { permanent_uuid: "3b28e6c9e0414b90b1fb162ad0b440d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42051 } }
I20260929 15:51:53.655635 5003 leader_election.cc:288] T 5d37624a8979474c923ad507aa8e8573 P fd1a77d0d9474f29840491fcb6d1e37c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4dc73fed3fc4840abdc1670d2ba0e90 (127.0.97.1:45985), 3b28e6c9e0414b90b1fb162ad0b440d7 (127.0.97.2:42051)
I20260929 15:51:53.660961 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:53.655949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36311 (local address 127.0.97.1:45985)
0929 15:51:53.656159 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:53.656164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:53.656183 (+ 19us) server_negotiation.cc:407] Connection header received
0929 15:51:53.656409 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:53.656416 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:53.656481 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:53.656583 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:51:53.656989 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:53.657801 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:53.659361 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:53.660102 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:53.660250 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:53.660348 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:53.660600 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 15:51:53.660718 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 15:51:53.660972 4979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:53.655803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45985 (local address 127.0.97.3:36311)
0929 15:51:53.656091 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:53.656112 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:53.656236 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:53.656603 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:53.656607 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:53.656652 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:51:53.656836 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:53.656844 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:53.657924 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:53.657932 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:53.659191 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:53.659202 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:53.660257 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:53.660270 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:53.660447 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:53.660470 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 15:51:53.660577 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 15:51:53.660703 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260929 15:51:53.661659 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:53.656153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45649 (local address 127.0.97.2:42051)
0929 15:51:53.656364 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:53.656369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:53.656872 (+ 503us) server_negotiation.cc:407] Connection header received
0929 15:51:53.657080 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:53.657090 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:53.657145 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:53.657240 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:51:53.657743 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:53.658440 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:53.660041 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:53.660778 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:53.660929 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:53.661019 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:53.661311 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 15:51:53.661455 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 15:51:53.661737 5004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:53.656034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42051 (local address 127.0.97.3:45649)
0929 15:51:53.656761 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:53.656776 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:53.656936 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:53.657253 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:53.657258 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:53.657280 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:51:53.657582 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:53.657590 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:53.658588 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:53.658593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:53.659883 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:53.659893 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:53.660989 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:53.660995 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:53.661155 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:53.661185 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 15:51:53.661274 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 15:51:53.661416 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":411,"threads_started":1}
I20260929 15:51:53.662120 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d37624a8979474c923ad507aa8e8573" candidate_uuid: "fd1a77d0d9474f29840491fcb6d1e37c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4dc73fed3fc4840abdc1670d2ba0e90" is_pre_election: true
I20260929 15:51:53.662707 4667 raft_consensus.cc:2454] T 5d37624a8979474c923ad507aa8e8573 P e4dc73fed3fc4840abdc1670d2ba0e90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd1a77d0d9474f29840491fcb6d1e37c in term 0.
I20260929 15:51:53.662746 4800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d37624a8979474c923ad507aa8e8573" candidate_uuid: "fd1a77d0d9474f29840491fcb6d1e37c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b28e6c9e0414b90b1fb162ad0b440d7" is_pre_election: true
I20260929 15:51:53.663214 4800 raft_consensus.cc:2454] T 5d37624a8979474c923ad507aa8e8573 P 3b28e6c9e0414b90b1fb162ad0b440d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd1a77d0d9474f29840491fcb6d1e37c in term 0.
I20260929 15:51:53.663404 4867 leader_election.cc:302] T 5d37624a8979474c923ad507aa8e8573 P fd1a77d0d9474f29840491fcb6d1e37c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e4dc73fed3fc4840abdc1670d2ba0e90, fd1a77d0d9474f29840491fcb6d1e37c; no voters:
I20260929 15:51:53.663785 5003 raft_consensus.cc:2781] T 5d37624a8979474c923ad507aa8e8573 P fd1a77d0d9474f29840491fcb6d1e37c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:51:53.663949 5003 raft_consensus.cc:482] T 5d37624a8979474c923ad507aa8e8573 P fd1a77d0d9474f29840491fcb6d1e37c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:51:53.664075 5003 raft_consensus.cc:3037] T 5d37624a8979474c923ad507aa8e8573 P fd1a77d0d9474f29840491fcb6d1e37c [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:53.666364 5003 raft_consensus.cc:504] T 5d37624a8979474c923ad507aa8e8573 P fd1a77d0d9474f29840491fcb6d1e37c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1a77d0d9474f29840491fcb6d1e37c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38021 } } peers { permanent_uuid: "e4dc73fed3fc4840abdc1670d2ba0e90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45985 } } peers { permanent_uuid: "3b28e6c9e0414b90b1fb162ad0b440d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42051 } }
I20260929 15:51:53.666991 5003 leader_election.cc:288] T 5d37624a8979474c923ad507aa8e8573 P fd1a77d0d9474f29840491fcb6d1e37c [CANDIDATE]: Term 1 election: Requested vote from peers e4dc73fed3fc4840abdc1670d2ba0e90 (127.0.97.1:45985), 3b28e6c9e0414b90b1fb162ad0b440d7 (127.0.97.2:42051)
I20260929 15:51:53.667354 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d37624a8979474c923ad507aa8e8573" candidate_uuid: "fd1a77d0d9474f29840491fcb6d1e37c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4dc73fed3fc4840abdc1670d2ba0e90"
I20260929 15:51:53.667568 4667 raft_consensus.cc:3037] T 5d37624a8979474c923ad507aa8e8573 P e4dc73fed3fc4840abdc1670d2ba0e90 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:53.667687 4800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d37624a8979474c923ad507aa8e8573" candidate_uuid: "fd1a77d0d9474f29840491fcb6d1e37c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b28e6c9e0414b90b1fb162ad0b440d7"
I20260929 15:51:53.667897 4800 raft_consensus.cc:3037] T 5d37624a8979474c923ad507aa8e8573 P 3b28e6c9e0414b90b1fb162ad0b440d7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:53.669816 4667 raft_consensus.cc:2454] T 5d37624a8979474c923ad507aa8e8573 P e4dc73fed3fc4840abdc1670d2ba0e90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd1a77d0d9474f29840491fcb6d1e37c in term 1.
I20260929 15:51:53.670163 4800 raft_consensus.cc:2454] T 5d37624a8979474c923ad507aa8e8573 P 3b28e6c9e0414b90b1fb162ad0b440d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd1a77d0d9474f29840491fcb6d1e37c in term 1.
I20260929 15:51:53.670398 4867 leader_election.cc:302] T 5d37624a8979474c923ad507aa8e8573 P fd1a77d0d9474f29840491fcb6d1e37c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e4dc73fed3fc4840abdc1670d2ba0e90, fd1a77d0d9474f29840491fcb6d1e37c; no voters:
I20260929 15:51:53.670696 5003 raft_consensus.cc:2781] T 5d37624a8979474c923ad507aa8e8573 P fd1a77d0d9474f29840491fcb6d1e37c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:51:53.671237 5003 raft_consensus.cc:686] T 5d37624a8979474c923ad507aa8e8573 P fd1a77d0d9474f29840491fcb6d1e37c [term 1 LEADER]: Becoming Leader. State: Replica: fd1a77d0d9474f29840491fcb6d1e37c, State: Running, Role: LEADER
I20260929 15:51:53.671767 5003 consensus_queue.cc:237] T 5d37624a8979474c923ad507aa8e8573 P fd1a77d0d9474f29840491fcb6d1e37c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1a77d0d9474f29840491fcb6d1e37c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38021 } } peers { permanent_uuid: "e4dc73fed3fc4840abdc1670d2ba0e90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45985 } } peers { permanent_uuid: "3b28e6c9e0414b90b1fb162ad0b440d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42051 } }
I20260929 15:51:53.675897 4523 catalog_manager.cc:5495] T 5d37624a8979474c923ad507aa8e8573 P fd1a77d0d9474f29840491fcb6d1e37c reported cstate change: term changed from 0 to 1, leader changed from <none> to fd1a77d0d9474f29840491fcb6d1e37c (127.0.97.3). New cstate: current_term: 1 leader_uuid: "fd1a77d0d9474f29840491fcb6d1e37c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1a77d0d9474f29840491fcb6d1e37c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4dc73fed3fc4840abdc1670d2ba0e90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b28e6c9e0414b90b1fb162ad0b440d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42051 } health_report { overall_health: UNKNOWN } } }
I20260929 15:51:53.706547 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:53.701432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60234 (local address 127.0.97.3:38021)
0929 15:51:53.701651 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:53.701657 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:53.701761 (+ 104us) server_negotiation.cc:407] Connection header received
0929 15:51:53.701961 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:53.701965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:53.702038 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:53.702159 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:51:53.702462 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:53.703137 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:53.704445 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:53.704872 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:53.704981 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:53.705478 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:51:53.705977 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:51:53.706101 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:53.706280 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 15:51:53.706352 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 15:51:53.714190 4667 raft_consensus.cc:1261] T 5d37624a8979474c923ad507aa8e8573 P e4dc73fed3fc4840abdc1670d2ba0e90 [term 1 FOLLOWER]: Refusing update from remote peer fd1a77d0d9474f29840491fcb6d1e37c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:51:53.714191 4800 raft_consensus.cc:1261] T 5d37624a8979474c923ad507aa8e8573 P 3b28e6c9e0414b90b1fb162ad0b440d7 [term 1 FOLLOWER]: Refusing update from remote peer fd1a77d0d9474f29840491fcb6d1e37c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:51:53.715215 5005 consensus_queue.cc:1035] T 5d37624a8979474c923ad507aa8e8573 P fd1a77d0d9474f29840491fcb6d1e37c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4dc73fed3fc4840abdc1670d2ba0e90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:51:53.715569 5003 consensus_queue.cc:1035] T 5d37624a8979474c923ad507aa8e8573 P fd1a77d0d9474f29840491fcb6d1e37c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b28e6c9e0414b90b1fb162ad0b440d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:51:53.727947 5010 mvcc.cc:205] Tried to move back new op lower bound from 7334695377763700736 to 7334695377603084288. Current Snapshot: MvccSnapshot[applied={T|T < 7334695377763700736}]
I20260929 15:51:53.739306 4524 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:51:53.739596 4524 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:51:53.742947 4524 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5107030e1ffb44459f410a880475f86e: Sending DeleteTablet for 3 replicas of tablet 5d37624a8979474c923ad507aa8e8573
I20260929 15:51:53.743953 4912 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d37624a8979474c923ad507aa8e8573 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:51:53 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60218
I20260929 15:51:53.744011 4647 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d37624a8979474c923ad507aa8e8573 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:51:53 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42692
I20260929 15:51:53.744297 4780 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d37624a8979474c923ad507aa8e8573 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:51:53 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35288
I20260929 15:51:53.746595 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 4585
I20260929 15:51:53.756553 5017 tablet_replica.cc:330] T 5d37624a8979474c923ad507aa8e8573 P fd1a77d0d9474f29840491fcb6d1e37c: stopping tablet replica
I20260929 15:51:53.757468 5019 tablet_replica.cc:330] T 5d37624a8979474c923ad507aa8e8573 P 3b28e6c9e0414b90b1fb162ad0b440d7: stopping tablet replica
I20260929 15:51:53.757920 5017 raft_consensus.cc:2229] T 5d37624a8979474c923ad507aa8e8573 P fd1a77d0d9474f29840491fcb6d1e37c [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:51:53.757911 5019 raft_consensus.cc:2229] T 5d37624a8979474c923ad507aa8e8573 P 3b28e6c9e0414b90b1fb162ad0b440d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:51:53.758499 5019 raft_consensus.cc:2258] T 5d37624a8979474c923ad507aa8e8573 P 3b28e6c9e0414b90b1fb162ad0b440d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:51:53.759303 5018 tablet_replica.cc:330] T 5d37624a8979474c923ad507aa8e8573 P e4dc73fed3fc4840abdc1670d2ba0e90: stopping tablet replica
I20260929 15:51:53.759264 5017 raft_consensus.cc:2258] T 5d37624a8979474c923ad507aa8e8573 P fd1a77d0d9474f29840491fcb6d1e37c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:51:53.759730 5018 raft_consensus.cc:2229] T 5d37624a8979474c923ad507aa8e8573 P e4dc73fed3fc4840abdc1670d2ba0e90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:51:53.760164 5018 raft_consensus.cc:2258] T 5d37624a8979474c923ad507aa8e8573 P e4dc73fed3fc4840abdc1670d2ba0e90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:51:53.761724 5017 ts_tablet_manager.cc:1905] T 5d37624a8979474c923ad507aa8e8573 P fd1a77d0d9474f29840491fcb6d1e37c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:51:53.762025 5019 ts_tablet_manager.cc:1905] T 5d37624a8979474c923ad507aa8e8573 P 3b28e6c9e0414b90b1fb162ad0b440d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:51:53.767474 5018 ts_tablet_manager.cc:1905] T 5d37624a8979474c923ad507aa8e8573 P e4dc73fed3fc4840abdc1670d2ba0e90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:51:53.769188 5017 ts_tablet_manager.cc:1918] T 5d37624a8979474c923ad507aa8e8573 P fd1a77d0d9474f29840491fcb6d1e37c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:51:53.769426 5017 log.cc:1196] T 5d37624a8979474c923ad507aa8e8573 P fd1a77d0d9474f29840491fcb6d1e37c: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-2/wal/wals/5d37624a8979474c923ad507aa8e8573
I20260929 15:51:53.770069 5017 ts_tablet_manager.cc:1939] T 5d37624a8979474c923ad507aa8e8573 P fd1a77d0d9474f29840491fcb6d1e37c: Deleting consensus metadata
Sep 29 15:51:53 dist-test-slave-r0ck krb5kdc[4480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 15:51:53.771916 4509 catalog_manager.cc:4841] TS fd1a77d0d9474f29840491fcb6d1e37c (127.0.97.3:38021): tablet 5d37624a8979474c923ad507aa8e8573 (table test-table [id=04e65b30aca84cca8f7009c47d808fe8]) successfully deleted
I20260929 15:51:53.773876 5018 ts_tablet_manager.cc:1918] T 5d37624a8979474c923ad507aa8e8573 P e4dc73fed3fc4840abdc1670d2ba0e90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:51:53.774114 5018 log.cc:1196] T 5d37624a8979474c923ad507aa8e8573 P e4dc73fed3fc4840abdc1670d2ba0e90: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-0/wal/wals/5d37624a8979474c923ad507aa8e8573
I20260929 15:51:53.774736 5018 ts_tablet_manager.cc:1939] T 5d37624a8979474c923ad507aa8e8573 P e4dc73fed3fc4840abdc1670d2ba0e90: Deleting consensus metadata
I20260929 15:51:53.776264 5019 ts_tablet_manager.cc:1918] T 5d37624a8979474c923ad507aa8e8573 P 3b28e6c9e0414b90b1fb162ad0b440d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:51:53.776489 5019 log.cc:1196] T 5d37624a8979474c923ad507aa8e8573 P 3b28e6c9e0414b90b1fb162ad0b440d7: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697075657932-388-0/minicluster-data/ts-1/wal/wals/5d37624a8979474c923ad507aa8e8573
I20260929 15:51:53.776491 4510 catalog_manager.cc:4841] TS e4dc73fed3fc4840abdc1670d2ba0e90 (127.0.97.1:45985): tablet 5d37624a8979474c923ad507aa8e8573 (table test-table [id=04e65b30aca84cca8f7009c47d808fe8]) successfully deleted
I20260929 15:51:53.777036 5019 ts_tablet_manager.cc:1939] T 5d37624a8979474c923ad507aa8e8573 P 3b28e6c9e0414b90b1fb162ad0b440d7: Deleting consensus metadata
I20260929 15:51:53.778677 4508 catalog_manager.cc:4841] TS 3b28e6c9e0414b90b1fb162ad0b440d7 (127.0.97.2:42051): tablet 5d37624a8979474c923ad507aa8e8573 (table test-table [id=04e65b30aca84cca8f7009c47d808fe8]) successfully deleted
I20260929 15:51:53.779664 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:53.747765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42696 (local address 127.0.97.1:45985)
0929 15:51:53.747994 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:53.748003 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:53.760786 (+ 12783us) server_negotiation.cc:407] Connection header received
0929 15:51:53.761225 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:53.761231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:53.761359 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:53.761437 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:53.763166 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:53.764292 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:53.766391 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:53.767079 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:53.767180 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:53.772434 (+ 5254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:53.772451 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:53.772471 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:53.772521 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:53.776425 (+ 3904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:53.777197 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:53.777204 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:53.777206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:53.777335 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:53.777750 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:53.777755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:53.777758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:53.778097 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:53.778417 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:53.779079 (+ 662us) server_negotiation.cc:299] Negotiation successful
0929 15:51:53.779297 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 15:51:53.780339 4707 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:51:53.872385 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 4585
I20260929 15:51:53.889232 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 4718
Sep 29 15:51:53 dist-test-slave-r0ck krb5kdc[4480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 15:51:53.905107 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:53.890048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35290 (local address 127.0.97.2:42051)
0929 15:51:53.890233 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:53.890239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:53.890258 (+ 19us) server_negotiation.cc:407] Connection header received
0929 15:51:53.890450 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:53.890462 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:53.890553 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:53.890661 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:53.891879 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:53.892586 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:53.893845 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:53.894235 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:53.894371 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:53.898563 (+ 4192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:53.898582 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:53.898603 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:53.898648 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:53.902304 (+ 3656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:53.902927 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:53.902936 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:53.902940 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:53.903082 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:53.903487 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:53.903492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:53.903496 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:53.903926 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:53.904232 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:53.904516 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 15:51:53.904747 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 15:51:53.905869 4840 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:51:53.994333 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 4718
I20260929 15:51:54.011804 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 4850
Sep 29 15:51:54 dist-test-slave-r0ck krb5kdc[4480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 15:51:54.027815 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:54.012688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60246 (local address 127.0.97.3:38021)
0929 15:51:54.012889 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:54.012896 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:54.012927 (+ 31us) server_negotiation.cc:407] Connection header received
0929 15:51:54.012990 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:54.012999 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:54.013085 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:54.013195 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:54.014472 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:54.015329 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:54.016556 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:54.017032 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:54.017189 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:54.021828 (+ 4639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:54.021839 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:54.021857 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:51:54.021895 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:54.024920 (+ 3025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:54.025725 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:54.025732 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:54.025735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:54.025876 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:51:54.026310 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:51:54.026315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:51:54.026320 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:51:54.026657 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:54.026986 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:54.027241 (+ 255us) server_negotiation.cc:299] Negotiation successful
0929 15:51:54.027470 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 15:51:54.028748 4972 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:51:54.145828 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 4850
I20260929 15:51:54.164330 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 4493
I20260929 15:51:54.165534 4554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:51:54.271720 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 4493
2026-09-29T15:51:54Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3988 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T15:51:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:51:54Z Disabled control of system clock
I20260929 15:51:54.336506 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45793
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:41729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 15:51:54.649365 5028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:51:54.649632 5028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:51:54.649715 5028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:51:54.657379 5028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:51:54.657476 5028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:51:54.657519 5028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:51:54.657577 5028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:51:54.657618 5028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:51:54.666327 5028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45793
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:51:54.667484 5028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:51:54.669090 5028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:51:54.680194 5033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:54.680229 5036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:54.680150 5034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:51:54.681141 5028 server_base.cc:1034] running on GCE node
I20260929 15:51:54.681931 5028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:51:54.683282 5028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:51:54.684486 5028 hybrid_clock.cc:648] HybridClock initialized: now 1790697114684445 us; error 40 us; skew 500 ppm
I20260929 15:51:54.684862 5028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:51:54.690836 5028 webserver.cc:460] Webserver started at http://127.0.97.62:43683/ using document root <none> and password file <none>
I20260929 15:51:54.691504 5028 fs_manager.cc:359] Metadata directory not provided
I20260929 15:51:54.691591 5028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:51:54.691926 5028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:51:54.694761 5028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "0ab8cf92e214431e8ef4b052a60bb1aa"
format_stamp: "Formatted at 2026-09-29 15:51:54 on dist-test-slave-r0ck"
server_key: "061f34142ac8648f3da19b807b046597"
server_key_iv: "b2192dc0556d16367c7554c55292f909"
server_key_version: "encryptionkey@0"
I20260929 15:51:54.695546 5028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "0ab8cf92e214431e8ef4b052a60bb1aa"
format_stamp: "Formatted at 2026-09-29 15:51:54 on dist-test-slave-r0ck"
server_key: "061f34142ac8648f3da19b807b046597"
server_key_iv: "b2192dc0556d16367c7554c55292f909"
server_key_version: "encryptionkey@0"
I20260929 15:51:54.700378 5028 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 15:51:54.703666 5042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:54.704875 5028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 15:51:54.705049 5028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "0ab8cf92e214431e8ef4b052a60bb1aa"
format_stamp: "Formatted at 2026-09-29 15:51:54 on dist-test-slave-r0ck"
server_key: "061f34142ac8648f3da19b807b046597"
server_key_iv: "b2192dc0556d16367c7554c55292f909"
server_key_version: "encryptionkey@0"
I20260929 15:51:54.705281 5028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:51:54.710630 5028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:51:54.711344 5028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:51:54.711645 5028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:51:54.728363 5028 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45793
I20260929 15:51:54.728401 5093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45793 every 8 connection(s)
I20260929 15:51:54.729669 5028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:51:54.732846 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 5028
I20260929 15:51:54.733031 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:51:54.733584 5094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:54.733846 388 external_mini_cluster.cc:1428] Setting key 2c351e3e00e24ea5178bb1aa512e4fbd
I20260929 15:51:54.742482 5097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:54.735856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49024 (local address 127.0.97.62:45793)
0929 15:51:54.736862 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:54.736872 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:54.736910 (+ 38us) server_negotiation.cc:407] Connection header received
0929 15:51:54.739180 (+ 2270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:54.739201 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:54.739587 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:54.739904 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:54.740960 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:54.740966 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:54.740978 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:51:54.740993 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:54.741054 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:51:54.741069 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:54.741228 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:54.741419 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 15:51:54.741670 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":402,"threads_started":1}
I20260929 15:51:54.745524 5094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ab8cf92e214431e8ef4b052a60bb1aa: Bootstrap starting.
I20260929 15:51:54.748178 5094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ab8cf92e214431e8ef4b052a60bb1aa: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:54.749112 5094 log.cc:824] T 00000000000000000000000000000000 P 0ab8cf92e214431e8ef4b052a60bb1aa: Log is configured to *not* fsync() on all Append() calls
I20260929 15:51:54.751469 5094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ab8cf92e214431e8ef4b052a60bb1aa: No bootstrap required, opened a new log
I20260929 15:51:54.758164 5094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ab8cf92e214431e8ef4b052a60bb1aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab8cf92e214431e8ef4b052a60bb1aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45793 } }
I20260929 15:51:54.758478 5094 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ab8cf92e214431e8ef4b052a60bb1aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:54.758589 5094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ab8cf92e214431e8ef4b052a60bb1aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ab8cf92e214431e8ef4b052a60bb1aa, State: Initialized, Role: FOLLOWER
I20260929 15:51:54.759104 5094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ab8cf92e214431e8ef4b052a60bb1aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab8cf92e214431e8ef4b052a60bb1aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45793 } }
I20260929 15:51:54.759327 5094 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ab8cf92e214431e8ef4b052a60bb1aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:51:54.759466 5094 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ab8cf92e214431e8ef4b052a60bb1aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:51:54.759632 5094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ab8cf92e214431e8ef4b052a60bb1aa [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:54.761620 5094 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ab8cf92e214431e8ef4b052a60bb1aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab8cf92e214431e8ef4b052a60bb1aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45793 } }
I20260929 15:51:54.762076 5094 leader_election.cc:302] T 00000000000000000000000000000000 P 0ab8cf92e214431e8ef4b052a60bb1aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ab8cf92e214431e8ef4b052a60bb1aa; no voters:
I20260929 15:51:54.762598 5094 leader_election.cc:288] T 00000000000000000000000000000000 P 0ab8cf92e214431e8ef4b052a60bb1aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:51:54.762734 5099 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ab8cf92e214431e8ef4b052a60bb1aa [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:51:54.763867 5099 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ab8cf92e214431e8ef4b052a60bb1aa [term 1 LEADER]: Becoming Leader. State: Replica: 0ab8cf92e214431e8ef4b052a60bb1aa, State: Running, Role: LEADER
I20260929 15:51:54.764370 5099 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ab8cf92e214431e8ef4b052a60bb1aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab8cf92e214431e8ef4b052a60bb1aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45793 } }
I20260929 15:51:54.764634 5094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ab8cf92e214431e8ef4b052a60bb1aa [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:51:54.767022 5100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ab8cf92e214431e8ef4b052a60bb1aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ab8cf92e214431e8ef4b052a60bb1aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab8cf92e214431e8ef4b052a60bb1aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45793 } } }
I20260929 15:51:54.767428 5100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ab8cf92e214431e8ef4b052a60bb1aa [sys.catalog]: This master's current role is: LEADER
I20260929 15:51:54.767349 5101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ab8cf92e214431e8ef4b052a60bb1aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ab8cf92e214431e8ef4b052a60bb1aa. Latest consensus state: current_term: 1 leader_uuid: "0ab8cf92e214431e8ef4b052a60bb1aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab8cf92e214431e8ef4b052a60bb1aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45793 } } }
I20260929 15:51:54.767647 5101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ab8cf92e214431e8ef4b052a60bb1aa [sys.catalog]: This master's current role is: LEADER
I20260929 15:51:54.769052 5108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:51:54.773152 5108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:51:54.780643 5108 catalog_manager.cc:1347] Generated new cluster ID: 47c16da792c04229ae9ce216e58772f4
I20260929 15:51:54.780766 5108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:51:54.791640 5108 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:51:54.792008 5108 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:51:54.800787 5108 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ab8cf92e214431e8ef4b052a60bb1aa: Generated new TSK 0
I20260929 15:51:54.801699 5108 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:51:54.814947 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45793
--builtin_ntp_servers=127.0.97.20:41729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 15:51:55.129540 5118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:51:55.129796 5118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:51:55.129882 5118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:51:55.136806 5118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:51:55.136891 5118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:51:55.136994 5118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:51:55.145540 5118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45793
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:51:55.146662 5118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:51:55.148279 5118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:51:55.161046 5123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:55.160985 5124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:55.161172 5126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:51:55.161906 5118 server_base.cc:1034] running on GCE node
I20260929 15:51:55.162480 5118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:51:55.163375 5118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:51:55.164608 5118 hybrid_clock.cc:648] HybridClock initialized: now 1790697115164564 us; error 50 us; skew 500 ppm
I20260929 15:51:55.164983 5118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:51:55.167307 5118 webserver.cc:460] Webserver started at http://127.0.97.1:34111/ using document root <none> and password file <none>
I20260929 15:51:55.167994 5118 fs_manager.cc:359] Metadata directory not provided
I20260929 15:51:55.168123 5118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:51:55.168445 5118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:51:55.171393 5118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-0/data/instance:
uuid: "2e85b04a255a4c8698f45285e2969121"
format_stamp: "Formatted at 2026-09-29 15:51:55 on dist-test-slave-r0ck"
server_key: "6406a6733c231d4d51e4896b749be77e"
server_key_iv: "110ef38195e92381629991e611b272bc"
server_key_version: "encryptionkey@0"
I20260929 15:51:55.172142 5118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "2e85b04a255a4c8698f45285e2969121"
format_stamp: "Formatted at 2026-09-29 15:51:55 on dist-test-slave-r0ck"
server_key: "6406a6733c231d4d51e4896b749be77e"
server_key_iv: "110ef38195e92381629991e611b272bc"
server_key_version: "encryptionkey@0"
I20260929 15:51:55.177027 5118 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 15:51:55.180330 5132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:55.181638 5118 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:51:55.181807 5118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-0/wal
uuid: "2e85b04a255a4c8698f45285e2969121"
format_stamp: "Formatted at 2026-09-29 15:51:55 on dist-test-slave-r0ck"
server_key: "6406a6733c231d4d51e4896b749be77e"
server_key_iv: "110ef38195e92381629991e611b272bc"
server_key_version: "encryptionkey@0"
I20260929 15:51:55.182040 5118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:51:55.187353 5118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:51:55.188104 5118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:51:55.188367 5118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:51:55.189257 5118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:51:55.190713 5118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:51:55.190835 5118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:55.190966 5118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:51:55.191049 5118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:55.223079 5118 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44729
I20260929 15:51:55.223130 5244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44729 every 8 connection(s)
I20260929 15:51:55.224349 5118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:51:55.231968 5097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:55.226709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50715 (local address 127.0.97.62:45793)
0929 15:51:55.226870 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:55.226875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:55.228133 (+ 1258us) server_negotiation.cc:407] Connection header received
0929 15:51:55.229862 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:55.229867 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:55.229939 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:55.230048 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:55.231252 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:55.231257 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:55.231261 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:51:55.231279 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:55.231319 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:51:55.231330 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:55.231446 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:55.231641 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 15:51:55.231705 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 15:51:55.232522 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 5118
I20260929 15:51:55.232729 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:51:55.232929 5247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:55.227121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45793 (local address 127.0.97.1:50715)
0929 15:51:55.227967 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:55.228007 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:55.229602 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:55.230261 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:55.230269 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:55.230360 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:51:55.230975 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:51:55.230978 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:51:55.231013 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:51:55.231088 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:51:55.231435 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:51:55.231444 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:51:55.231452 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 15:51:55.231718 (+ 266us) client_negotiation.cc:240] Negotiation successful
0929 15:51:55.231951 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":270,"threads_started":1}
I20260929 15:51:55.233502 388 external_mini_cluster.cc:1428] Setting key 4e2c8c59160937677bcea3415eb1cd54
I20260929 15:51:55.234788 5245 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45793
I20260929 15:51:55.235075 5245 heartbeater.cc:464] Registering TS with master...
I20260929 15:51:55.235718 5245 heartbeater.cc:511] Master 127.0.97.62:45793 requested a full tablet report, sending...
I20260929 15:51:55.236498 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45793
--builtin_ntp_servers=127.0.97.20:41729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 15:51:55.238031 5059 ts_manager.cc:194] Registered new tserver with Master: 2e85b04a255a4c8698f45285e2969121 (127.0.97.1:44729)
W20260929 15:51:55.549674 5248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:51:55.549981 5248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:51:55.550143 5248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:51:55.557308 5248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:51:55.557497 5248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:51:55.557674 5248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:51:55.566756 5248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45793
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:51:55.567934 5248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:51:55.569679 5248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:51:55.582345 5253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:55.582396 5254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:55.582427 5256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:51:55.583730 5248 server_base.cc:1034] running on GCE node
I20260929 15:51:55.584295 5248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:51:55.585170 5248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:51:55.586402 5248 hybrid_clock.cc:648] HybridClock initialized: now 1790697115586357 us; error 54 us; skew 500 ppm
I20260929 15:51:55.586722 5248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:51:55.588908 5248 webserver.cc:460] Webserver started at http://127.0.97.2:41411/ using document root <none> and password file <none>
I20260929 15:51:55.589649 5248 fs_manager.cc:359] Metadata directory not provided
I20260929 15:51:55.589778 5248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:51:55.590097 5248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:51:55.592989 5248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-1/data/instance:
uuid: "1027d5f8e9774183b38a370e3312a2e3"
format_stamp: "Formatted at 2026-09-29 15:51:55 on dist-test-slave-r0ck"
server_key: "432e350aaff1bd7ce2696004a486dd29"
server_key_iv: "fcae40ec054bd5c726e9f5e3ef136835"
server_key_version: "encryptionkey@0"
I20260929 15:51:55.593920 5248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1027d5f8e9774183b38a370e3312a2e3"
format_stamp: "Formatted at 2026-09-29 15:51:55 on dist-test-slave-r0ck"
server_key: "432e350aaff1bd7ce2696004a486dd29"
server_key_iv: "fcae40ec054bd5c726e9f5e3ef136835"
server_key_version: "encryptionkey@0"
I20260929 15:51:55.598726 5248 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 15:51:55.601979 5262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:55.603130 5248 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 15:51:55.603305 5248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-1/wal
uuid: "1027d5f8e9774183b38a370e3312a2e3"
format_stamp: "Formatted at 2026-09-29 15:51:55 on dist-test-slave-r0ck"
server_key: "432e350aaff1bd7ce2696004a486dd29"
server_key_iv: "fcae40ec054bd5c726e9f5e3ef136835"
server_key_version: "encryptionkey@0"
I20260929 15:51:55.603523 5248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:51:55.609042 5248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:51:55.609867 5248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:51:55.610136 5248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:51:55.611025 5248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:51:55.612375 5248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:51:55.612496 5248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:55.612635 5248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:51:55.612723 5248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:55.646409 5248 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34589
I20260929 15:51:55.646498 5374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34589 every 8 connection(s)
I20260929 15:51:55.647718 5248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:51:55.655018 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 5248
I20260929 15:51:55.655198 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:51:55.655485 5097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:55.650230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60345 (local address 127.0.97.62:45793)
0929 15:51:55.650446 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:55.650452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:55.651793 (+ 1341us) server_negotiation.cc:407] Connection header received
0929 15:51:55.653436 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:55.653440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:55.653491 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:55.653572 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:55.654740 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:55.654744 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:55.654747 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:51:55.654761 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:55.654788 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:51:55.654799 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:55.654932 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:55.655120 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 15:51:55.655198 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 15:51:55.655699 388 external_mini_cluster.cc:1428] Setting key 69041f2085db9756c8434a2e8eacf703
I20260929 15:51:55.656421 5377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:55.650755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45793 (local address 127.0.97.2:60345)
0929 15:51:55.651649 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:55.651689 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:55.653124 (+ 1435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:55.653762 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:55.653769 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:55.653854 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:51:55.654466 (+ 612us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:51:55.654468 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:51:55.654503 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:51:55.654579 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:51:55.654909 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:51:55.654919 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:51:55.654927 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 15:51:55.655199 (+ 272us) client_negotiation.cc:240] Negotiation successful
0929 15:51:55.655425 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":261,"threads_started":1}
I20260929 15:51:55.657779 5375 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45793
I20260929 15:51:55.658051 5375 heartbeater.cc:464] Registering TS with master...
I20260929 15:51:55.658558 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45793
--builtin_ntp_servers=127.0.97.20:41729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 15:51:55.658631 5375 heartbeater.cc:511] Master 127.0.97.62:45793 requested a full tablet report, sending...
I20260929 15:51:55.659950 5058 ts_manager.cc:194] Registered new tserver with Master: 1027d5f8e9774183b38a370e3312a2e3 (127.0.97.2:34589)
W20260929 15:51:55.978221 5378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:51:55.978479 5378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:51:55.978642 5378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:51:55.985883 5378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:51:55.985975 5378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:51:55.986133 5378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:51:55.995177 5378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45793
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:51:55.996367 5378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:51:55.998085 5378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:51:56.012049 5383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:56.012004 5386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:56.012097 5384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:51:56.013562 5378 server_base.cc:1034] running on GCE node
I20260929 15:51:56.014206 5378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:51:56.015183 5378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:51:56.016443 5378 hybrid_clock.cc:648] HybridClock initialized: now 1790697116016395 us; error 58 us; skew 500 ppm
I20260929 15:51:56.016808 5378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:51:56.019292 5378 webserver.cc:460] Webserver started at http://127.0.97.3:36581/ using document root <none> and password file <none>
I20260929 15:51:56.019986 5378 fs_manager.cc:359] Metadata directory not provided
I20260929 15:51:56.020130 5378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:51:56.020489 5378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:51:56.023391 5378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-2/data/instance:
uuid: "d6d8a4e8ae7e492ab63f7ea9d8f076f3"
format_stamp: "Formatted at 2026-09-29 15:51:56 on dist-test-slave-r0ck"
server_key: "eb67fdf6f0845c788cf7e0270a698986"
server_key_iv: "1bdc10130ed0f99aac0348c248e6c910"
server_key_version: "encryptionkey@0"
I20260929 15:51:56.024192 5378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d6d8a4e8ae7e492ab63f7ea9d8f076f3"
format_stamp: "Formatted at 2026-09-29 15:51:56 on dist-test-slave-r0ck"
server_key: "eb67fdf6f0845c788cf7e0270a698986"
server_key_iv: "1bdc10130ed0f99aac0348c248e6c910"
server_key_version: "encryptionkey@0"
I20260929 15:51:56.029158 5378 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260929 15:51:56.032769 5392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:56.034165 5378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:51:56.034354 5378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-2/wal
uuid: "d6d8a4e8ae7e492ab63f7ea9d8f076f3"
format_stamp: "Formatted at 2026-09-29 15:51:56 on dist-test-slave-r0ck"
server_key: "eb67fdf6f0845c788cf7e0270a698986"
server_key_iv: "1bdc10130ed0f99aac0348c248e6c910"
server_key_version: "encryptionkey@0"
I20260929 15:51:56.034598 5378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:51:56.039994 5378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:51:56.040736 5378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:51:56.040995 5378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:51:56.041966 5378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:51:56.043298 5378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:51:56.043427 5378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:56.043560 5378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:51:56.043651 5378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:56.077596 5378 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42867
I20260929 15:51:56.077633 5504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42867 every 8 connection(s)
I20260929 15:51:56.078946 5378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:51:56.087206 5097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:56.081691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56739 (local address 127.0.97.62:45793)
0929 15:51:56.081891 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:56.081896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:56.083278 (+ 1382us) server_negotiation.cc:407] Connection header received
0929 15:51:56.084863 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:56.084867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:56.084919 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:56.085012 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:56.086421 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:56.086426 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:56.086429 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:51:56.086472 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:56.086518 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:51:56.086532 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:56.086641 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:56.086944 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 15:51:56.086997 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 15:51:56.087260 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 5378
I20260929 15:51:56.087422 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:51:56.087984 388 external_mini_cluster.cc:1428] Setting key c14dd7dcdaae7652a6ddca0d2043a3ac
I20260929 15:51:56.088495 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:56.082235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45793 (local address 127.0.97.3:56739)
0929 15:51:56.083128 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:56.083169 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:56.084622 (+ 1453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:56.085249 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:56.085257 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:56.085412 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:51:56.086119 (+ 707us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:51:56.086122 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:51:56.086158 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:51:56.086237 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:51:56.086658 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:51:56.086666 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:51:56.086673 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 15:51:56.086991 (+ 318us) client_negotiation.cc:240] Negotiation successful
0929 15:51:56.087264 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":294,"threads_started":1}
I20260929 15:51:56.090042 5505 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45793
I20260929 15:51:56.090372 5505 heartbeater.cc:464] Registering TS with master...
I20260929 15:51:56.091034 5505 heartbeater.cc:511] Master 127.0.97.62:45793 requested a full tablet report, sending...
I20260929 15:51:56.092412 5058 ts_manager.cc:194] Registered new tserver with Master: d6d8a4e8ae7e492ab63f7ea9d8f076f3 (127.0.97.3:42867)
I20260929 15:51:56.103852 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:51:56.110649 5097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:56.108378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49038 (local address 127.0.97.62:45793)
0929 15:51:56.108536 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:56.108541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:56.109023 (+ 482us) server_negotiation.cc:407] Connection header received
0929 15:51:56.109154 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:56.109158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:56.109204 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:56.109286 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:56.110162 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:56.110166 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:56.110170 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:51:56.110184 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:56.110212 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:51:56.110223 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:56.110317 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:56.110402 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 15:51:56.110441 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 15:51:56.114569 5058 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49038:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:51:56.121044 5058 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:51:56.149468 5517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:56.132499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42867 (local address 127.0.0.1:46710)
0929 15:51:56.145824 (+ 13325us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:56.145845 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:56.145952 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:56.146839 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:56.146846 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:56.146947 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:51:56.147721 (+ 774us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:51:56.147726 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:51:56.147758 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:51:56.147842 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:51:56.148442 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:51:56.148452 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:51:56.148459 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 15:51:56.148919 (+ 460us) client_negotiation.cc:240] Negotiation successful
0929 15:51:56.149008 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13168,"thread_start_us":11522,"threads_started":1}
I20260929 15:51:56.150252 5520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:56.133010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46710 (local address 127.0.97.3:42867)
0929 15:51:56.145450 (+ 12440us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:56.145461 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:56.145953 (+ 492us) server_negotiation.cc:407] Connection header received
0929 15:51:56.146094 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:56.146105 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:56.146350 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:56.146511 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:56.148215 (+ 1704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:56.148226 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:56.148242 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:51:56.148269 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:56.148323 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:51:56.148347 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:56.148514 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:56.148648 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 15:51:56.148747 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12216,"thread_start_us":12459,"threads_started":1}
I20260929 15:51:56.150286 5518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:56.132267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44729 (local address 127.0.0.1:57142)
0929 15:51:56.144394 (+ 12127us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:56.144430 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:56.144560 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:56.147720 (+ 3160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:56.147725 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:56.147752 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:51:56.148437 (+ 685us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:51:56.148441 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:51:56.148459 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:51:56.148507 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:51:56.149597 (+ 1090us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:51:56.149601 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:51:56.149606 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 15:51:56.149734 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 15:51:56.149790 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11896,"thread_start_us":13254,"threads_started":1}
I20260929 15:51:56.150362 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:56.133096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34589 (local address 127.0.0.1:36710)
0929 15:51:56.146367 (+ 13271us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:56.146379 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:56.146485 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:56.148289 (+ 1804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:56.148293 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:56.148314 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:51:56.148987 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:51:56.148992 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:51:56.149008 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:51:56.149054 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:51:56.149657 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:51:56.149662 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:51:56.149666 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 15:51:56.149816 (+ 150us) client_negotiation.cc:240] Negotiation successful
0929 15:51:56.149858 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13182,"thread_start_us":13084,"threads_started":1}
I20260929 15:51:56.151046 5519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:56.132507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57142 (local address 127.0.97.1:44729)
0929 15:51:56.146959 (+ 14452us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:56.146966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:56.146990 (+ 24us) server_negotiation.cc:407] Connection header received
0929 15:51:56.147045 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:56.147051 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:56.147198 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:56.148152 (+ 954us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:56.149381 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:56.149388 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:56.149397 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:51:56.149419 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:56.149460 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:51:56.149479 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:56.150611 (+ 1132us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:56.150673 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 15:51:56.150741 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14298,"thread_start_us":14180,"threads_started":1}
I20260929 15:51:56.151090 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:56.133824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36710 (local address 127.0.97.2:34589)
0929 15:51:56.146080 (+ 12256us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:56.146086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:56.146470 (+ 384us) server_negotiation.cc:407] Connection header received
0929 15:51:56.146965 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:56.146974 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:56.147153 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:56.147293 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:56.149219 (+ 1926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:56.149320 (+ 101us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:56.149334 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:51:56.149377 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:56.149430 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:51:56.149451 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:56.149609 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:56.150640 (+ 1031us) server_negotiation.cc:299] Negotiation successful
0929 15:51:56.150732 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12084,"thread_start_us":11935,"threads_started":1}
I20260929 15:51:56.153980 5440 tablet_service.cc:1467] Processing CreateTablet for tablet e1e4def7afd9419ea98dfbf299777406 (DEFAULT_TABLE table=test-table [id=0deb3a2d5b8740b69a8408ef5850726f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:51:56.154206 5180 tablet_service.cc:1467] Processing CreateTablet for tablet e1e4def7afd9419ea98dfbf299777406 (DEFAULT_TABLE table=test-table [id=0deb3a2d5b8740b69a8408ef5850726f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:51:56.154363 5310 tablet_service.cc:1467] Processing CreateTablet for tablet e1e4def7afd9419ea98dfbf299777406 (DEFAULT_TABLE table=test-table [id=0deb3a2d5b8740b69a8408ef5850726f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:51:56.155329 5180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1e4def7afd9419ea98dfbf299777406. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:56.155395 5440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1e4def7afd9419ea98dfbf299777406. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:56.155539 5310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1e4def7afd9419ea98dfbf299777406. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:56.166357 5523 tablet_bootstrap.cc:492] T e1e4def7afd9419ea98dfbf299777406 P 2e85b04a255a4c8698f45285e2969121: Bootstrap starting.
I20260929 15:51:56.170372 5523 tablet_bootstrap.cc:654] T e1e4def7afd9419ea98dfbf299777406 P 2e85b04a255a4c8698f45285e2969121: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:56.170514 5524 tablet_bootstrap.cc:492] T e1e4def7afd9419ea98dfbf299777406 P 1027d5f8e9774183b38a370e3312a2e3: Bootstrap starting.
I20260929 15:51:56.171006 5525 tablet_bootstrap.cc:492] T e1e4def7afd9419ea98dfbf299777406 P d6d8a4e8ae7e492ab63f7ea9d8f076f3: Bootstrap starting.
I20260929 15:51:56.172760 5523 log.cc:824] T e1e4def7afd9419ea98dfbf299777406 P 2e85b04a255a4c8698f45285e2969121: Log is configured to *not* fsync() on all Append() calls
I20260929 15:51:56.174533 5525 tablet_bootstrap.cc:654] T e1e4def7afd9419ea98dfbf299777406 P d6d8a4e8ae7e492ab63f7ea9d8f076f3: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:56.175699 5525 log.cc:824] T e1e4def7afd9419ea98dfbf299777406 P d6d8a4e8ae7e492ab63f7ea9d8f076f3: Log is configured to *not* fsync() on all Append() calls
I20260929 15:51:56.176070 5524 tablet_bootstrap.cc:654] T e1e4def7afd9419ea98dfbf299777406 P 1027d5f8e9774183b38a370e3312a2e3: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:56.176869 5523 tablet_bootstrap.cc:492] T e1e4def7afd9419ea98dfbf299777406 P 2e85b04a255a4c8698f45285e2969121: No bootstrap required, opened a new log
I20260929 15:51:56.176986 5524 log.cc:824] T e1e4def7afd9419ea98dfbf299777406 P 1027d5f8e9774183b38a370e3312a2e3: Log is configured to *not* fsync() on all Append() calls
I20260929 15:51:56.177193 5523 ts_tablet_manager.cc:1397] T e1e4def7afd9419ea98dfbf299777406 P 2e85b04a255a4c8698f45285e2969121: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20260929 15:51:56.179958 5525 tablet_bootstrap.cc:492] T e1e4def7afd9419ea98dfbf299777406 P d6d8a4e8ae7e492ab63f7ea9d8f076f3: No bootstrap required, opened a new log
I20260929 15:51:56.180186 5525 ts_tablet_manager.cc:1397] T e1e4def7afd9419ea98dfbf299777406 P d6d8a4e8ae7e492ab63f7ea9d8f076f3: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260929 15:51:56.183174 5524 tablet_bootstrap.cc:492] T e1e4def7afd9419ea98dfbf299777406 P 1027d5f8e9774183b38a370e3312a2e3: No bootstrap required, opened a new log
I20260929 15:51:56.183429 5524 ts_tablet_manager.cc:1397] T e1e4def7afd9419ea98dfbf299777406 P 1027d5f8e9774183b38a370e3312a2e3: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260929 15:51:56.187691 5523 raft_consensus.cc:348] T e1e4def7afd9419ea98dfbf299777406 P 2e85b04a255a4c8698f45285e2969121 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e85b04a255a4c8698f45285e2969121" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44729 } } peers { permanent_uuid: "d6d8a4e8ae7e492ab63f7ea9d8f076f3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42867 } } peers { permanent_uuid: "1027d5f8e9774183b38a370e3312a2e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34589 } }
I20260929 15:51:56.188122 5523 raft_consensus.cc:374] T e1e4def7afd9419ea98dfbf299777406 P 2e85b04a255a4c8698f45285e2969121 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:56.188254 5523 raft_consensus.cc:729] T e1e4def7afd9419ea98dfbf299777406 P 2e85b04a255a4c8698f45285e2969121 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e85b04a255a4c8698f45285e2969121, State: Initialized, Role: FOLLOWER
I20260929 15:51:56.188892 5523 consensus_queue.cc:260] T e1e4def7afd9419ea98dfbf299777406 P 2e85b04a255a4c8698f45285e2969121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e85b04a255a4c8698f45285e2969121" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44729 } } peers { permanent_uuid: "d6d8a4e8ae7e492ab63f7ea9d8f076f3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42867 } } peers { permanent_uuid: "1027d5f8e9774183b38a370e3312a2e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34589 } }
I20260929 15:51:56.190024 5245 heartbeater.cc:503] Master 127.0.97.62:45793 was elected leader, sending a full tablet report...
I20260929 15:51:56.190661 5523 ts_tablet_manager.cc:1428] T e1e4def7afd9419ea98dfbf299777406 P 2e85b04a255a4c8698f45285e2969121: Time spent starting tablet: real 0.013s user 0.012s sys 0.003s
I20260929 15:51:56.190377 5525 raft_consensus.cc:348] T e1e4def7afd9419ea98dfbf299777406 P d6d8a4e8ae7e492ab63f7ea9d8f076f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e85b04a255a4c8698f45285e2969121" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44729 } } peers { permanent_uuid: "d6d8a4e8ae7e492ab63f7ea9d8f076f3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42867 } } peers { permanent_uuid: "1027d5f8e9774183b38a370e3312a2e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34589 } }
I20260929 15:51:56.190793 5525 raft_consensus.cc:374] T e1e4def7afd9419ea98dfbf299777406 P d6d8a4e8ae7e492ab63f7ea9d8f076f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:56.190891 5525 raft_consensus.cc:729] T e1e4def7afd9419ea98dfbf299777406 P d6d8a4e8ae7e492ab63f7ea9d8f076f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6d8a4e8ae7e492ab63f7ea9d8f076f3, State: Initialized, Role: FOLLOWER
I20260929 15:51:56.191485 5525 consensus_queue.cc:260] T e1e4def7afd9419ea98dfbf299777406 P d6d8a4e8ae7e492ab63f7ea9d8f076f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e85b04a255a4c8698f45285e2969121" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44729 } } peers { permanent_uuid: "d6d8a4e8ae7e492ab63f7ea9d8f076f3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42867 } } peers { permanent_uuid: "1027d5f8e9774183b38a370e3312a2e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34589 } }
I20260929 15:51:56.193099 5525 ts_tablet_manager.cc:1428] T e1e4def7afd9419ea98dfbf299777406 P d6d8a4e8ae7e492ab63f7ea9d8f076f3: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260929 15:51:56.193236 5524 raft_consensus.cc:348] T e1e4def7afd9419ea98dfbf299777406 P 1027d5f8e9774183b38a370e3312a2e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e85b04a255a4c8698f45285e2969121" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44729 } } peers { permanent_uuid: "d6d8a4e8ae7e492ab63f7ea9d8f076f3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42867 } } peers { permanent_uuid: "1027d5f8e9774183b38a370e3312a2e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34589 } }
I20260929 15:51:56.193634 5524 raft_consensus.cc:374] T e1e4def7afd9419ea98dfbf299777406 P 1027d5f8e9774183b38a370e3312a2e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:56.193718 5505 heartbeater.cc:503] Master 127.0.97.62:45793 was elected leader, sending a full tablet report...
I20260929 15:51:56.193740 5524 raft_consensus.cc:729] T e1e4def7afd9419ea98dfbf299777406 P 1027d5f8e9774183b38a370e3312a2e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1027d5f8e9774183b38a370e3312a2e3, State: Initialized, Role: FOLLOWER
I20260929 15:51:56.194312 5524 consensus_queue.cc:260] T e1e4def7afd9419ea98dfbf299777406 P 1027d5f8e9774183b38a370e3312a2e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e85b04a255a4c8698f45285e2969121" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44729 } } peers { permanent_uuid: "d6d8a4e8ae7e492ab63f7ea9d8f076f3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42867 } } peers { permanent_uuid: "1027d5f8e9774183b38a370e3312a2e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34589 } }
I20260929 15:51:56.197999 5375 heartbeater.cc:503] Master 127.0.97.62:45793 was elected leader, sending a full tablet report...
I20260929 15:51:56.198504 5524 ts_tablet_manager.cc:1428] T e1e4def7afd9419ea98dfbf299777406 P 1027d5f8e9774183b38a370e3312a2e3: Time spent starting tablet: real 0.015s user 0.010s sys 0.003s
W20260929 15:51:56.226532 5246 tablet.cc:2369] T e1e4def7afd9419ea98dfbf299777406 P 2e85b04a255a4c8698f45285e2969121: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:51:56.330633 5506 tablet.cc:2369] T e1e4def7afd9419ea98dfbf299777406 P d6d8a4e8ae7e492ab63f7ea9d8f076f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:51:56.397611 5529 raft_consensus.cc:482] T e1e4def7afd9419ea98dfbf299777406 P 2e85b04a255a4c8698f45285e2969121 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:51:56.397902 5529 raft_consensus.cc:504] T e1e4def7afd9419ea98dfbf299777406 P 2e85b04a255a4c8698f45285e2969121 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e85b04a255a4c8698f45285e2969121" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44729 } } peers { permanent_uuid: "d6d8a4e8ae7e492ab63f7ea9d8f076f3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42867 } } peers { permanent_uuid: "1027d5f8e9774183b38a370e3312a2e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34589 } }
I20260929 15:51:56.399621 5529 leader_election.cc:288] T e1e4def7afd9419ea98dfbf299777406 P 2e85b04a255a4c8698f45285e2969121 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6d8a4e8ae7e492ab63f7ea9d8f076f3 (127.0.97.3:42867), 1027d5f8e9774183b38a370e3312a2e3 (127.0.97.2:34589)
W20260929 15:51:56.399698 5376 tablet.cc:2369] T e1e4def7afd9419ea98dfbf299777406 P 1027d5f8e9774183b38a370e3312a2e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:51:56.403251 5520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:56.400340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41547 (local address 127.0.97.3:42867)
0929 15:51:56.400539 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:56.400546 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:56.401200 (+ 654us) server_negotiation.cc:407] Connection header received
0929 15:51:56.401429 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:56.401433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:56.401481 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:56.401581 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:56.402581 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:56.402585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:56.402588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:51:56.402604 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:56.402633 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:51:56.402678 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:56.402805 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:56.402925 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 15:51:56.402990 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 15:51:56.403324 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:56.400366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40223 (local address 127.0.97.2:34589)
0929 15:51:56.400611 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:56.400620 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:56.401330 (+ 710us) server_negotiation.cc:407] Connection header received
0929 15:51:56.401527 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:56.401537 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:56.401586 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:56.401686 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:56.402729 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:56.402734 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:56.402739 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:51:56.402756 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:56.402791 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:51:56.402802 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:56.402932 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:56.403008 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 15:51:56.403067 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 15:51:56.403470 5532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:56.399879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42867 (local address 127.0.97.1:41547)
0929 15:51:56.401085 (+ 1206us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:56.401104 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:56.401251 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:56.401602 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:56.401608 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:56.401632 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:51:56.402353 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:51:56.402356 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:51:56.402370 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:51:56.402406 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:51:56.402786 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:51:56.402791 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:51:56.402797 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 15:51:56.402956 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 15:51:56.403002 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1089,"thread_start_us":879,"threads_started":1}
I20260929 15:51:56.403579 5533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:56.400067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34589 (local address 127.0.97.1:40223)
0929 15:51:56.401229 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:56.401243 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:56.401391 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:56.401711 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:56.401716 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:56.401739 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:51:56.402549 (+ 810us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:51:56.402551 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:51:56.402564 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:51:56.402596 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:51:56.402901 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:51:56.402905 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:51:56.402911 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 15:51:56.403058 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 15:51:56.403102 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1067,"thread_start_us":785,"threads_started":1}
I20260929 15:51:56.404475 5460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1e4def7afd9419ea98dfbf299777406" candidate_uuid: "2e85b04a255a4c8698f45285e2969121" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6d8a4e8ae7e492ab63f7ea9d8f076f3" is_pre_election: true
I20260929 15:51:56.404464 5330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1e4def7afd9419ea98dfbf299777406" candidate_uuid: "2e85b04a255a4c8698f45285e2969121" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1027d5f8e9774183b38a370e3312a2e3" is_pre_election: true
I20260929 15:51:56.404919 5330 raft_consensus.cc:2454] T e1e4def7afd9419ea98dfbf299777406 P 1027d5f8e9774183b38a370e3312a2e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e85b04a255a4c8698f45285e2969121 in term 0.
I20260929 15:51:56.404925 5460 raft_consensus.cc:2454] T e1e4def7afd9419ea98dfbf299777406 P d6d8a4e8ae7e492ab63f7ea9d8f076f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e85b04a255a4c8698f45285e2969121 in term 0.
I20260929 15:51:56.405591 5136 leader_election.cc:302] T e1e4def7afd9419ea98dfbf299777406 P 2e85b04a255a4c8698f45285e2969121 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e85b04a255a4c8698f45285e2969121, d6d8a4e8ae7e492ab63f7ea9d8f076f3; no voters:
I20260929 15:51:56.405975 5529 raft_consensus.cc:2781] T e1e4def7afd9419ea98dfbf299777406 P 2e85b04a255a4c8698f45285e2969121 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:51:56.406113 5529 raft_consensus.cc:482] T e1e4def7afd9419ea98dfbf299777406 P 2e85b04a255a4c8698f45285e2969121 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:51:56.406217 5529 raft_consensus.cc:3037] T e1e4def7afd9419ea98dfbf299777406 P 2e85b04a255a4c8698f45285e2969121 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:56.408643 5529 raft_consensus.cc:504] T e1e4def7afd9419ea98dfbf299777406 P 2e85b04a255a4c8698f45285e2969121 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e85b04a255a4c8698f45285e2969121" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44729 } } peers { permanent_uuid: "d6d8a4e8ae7e492ab63f7ea9d8f076f3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42867 } } peers { permanent_uuid: "1027d5f8e9774183b38a370e3312a2e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34589 } }
I20260929 15:51:56.409276 5529 leader_election.cc:288] T e1e4def7afd9419ea98dfbf299777406 P 2e85b04a255a4c8698f45285e2969121 [CANDIDATE]: Term 1 election: Requested vote from peers d6d8a4e8ae7e492ab63f7ea9d8f076f3 (127.0.97.3:42867), 1027d5f8e9774183b38a370e3312a2e3 (127.0.97.2:34589)
I20260929 15:51:56.409711 5460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1e4def7afd9419ea98dfbf299777406" candidate_uuid: "2e85b04a255a4c8698f45285e2969121" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6d8a4e8ae7e492ab63f7ea9d8f076f3"
I20260929 15:51:56.409909 5460 raft_consensus.cc:3037] T e1e4def7afd9419ea98dfbf299777406 P d6d8a4e8ae7e492ab63f7ea9d8f076f3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:56.411971 5460 raft_consensus.cc:2454] T e1e4def7afd9419ea98dfbf299777406 P d6d8a4e8ae7e492ab63f7ea9d8f076f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e85b04a255a4c8698f45285e2969121 in term 1.
I20260929 15:51:56.412272 5330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1e4def7afd9419ea98dfbf299777406" candidate_uuid: "2e85b04a255a4c8698f45285e2969121" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1027d5f8e9774183b38a370e3312a2e3"
I20260929 15:51:56.412406 5136 leader_election.cc:302] T e1e4def7afd9419ea98dfbf299777406 P 2e85b04a255a4c8698f45285e2969121 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e85b04a255a4c8698f45285e2969121, d6d8a4e8ae7e492ab63f7ea9d8f076f3; no voters:
I20260929 15:51:56.412456 5330 raft_consensus.cc:3037] T e1e4def7afd9419ea98dfbf299777406 P 1027d5f8e9774183b38a370e3312a2e3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:56.412699 5529 raft_consensus.cc:2781] T e1e4def7afd9419ea98dfbf299777406 P 2e85b04a255a4c8698f45285e2969121 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:51:56.413218 5529 raft_consensus.cc:686] T e1e4def7afd9419ea98dfbf299777406 P 2e85b04a255a4c8698f45285e2969121 [term 1 LEADER]: Becoming Leader. State: Replica: 2e85b04a255a4c8698f45285e2969121, State: Running, Role: LEADER
I20260929 15:51:56.413851 5529 consensus_queue.cc:237] T e1e4def7afd9419ea98dfbf299777406 P 2e85b04a255a4c8698f45285e2969121 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e85b04a255a4c8698f45285e2969121" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44729 } } peers { permanent_uuid: "d6d8a4e8ae7e492ab63f7ea9d8f076f3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42867 } } peers { permanent_uuid: "1027d5f8e9774183b38a370e3312a2e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34589 } }
I20260929 15:51:56.415593 5330 raft_consensus.cc:2454] T e1e4def7afd9419ea98dfbf299777406 P 1027d5f8e9774183b38a370e3312a2e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e85b04a255a4c8698f45285e2969121 in term 1.
I20260929 15:51:56.418958 5059 catalog_manager.cc:5495] T e1e4def7afd9419ea98dfbf299777406 P 2e85b04a255a4c8698f45285e2969121 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e85b04a255a4c8698f45285e2969121 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "2e85b04a255a4c8698f45285e2969121" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e85b04a255a4c8698f45285e2969121" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6d8a4e8ae7e492ab63f7ea9d8f076f3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1027d5f8e9774183b38a370e3312a2e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34589 } health_report { overall_health: UNKNOWN } } }
I20260929 15:51:56.445508 5519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:56.443614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57150 (local address 127.0.97.1:44729)
0929 15:51:56.443771 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:56.443775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:56.443894 (+ 119us) server_negotiation.cc:407] Connection header received
0929 15:51:56.444043 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:56.444047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:56.444122 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:56.444203 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:56.444975 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:56.444979 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:56.444982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:51:56.444996 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:56.445024 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:51:56.445034 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:56.445121 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:56.445192 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 15:51:56.445232 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 15:51:56.452863 5460 raft_consensus.cc:1261] T e1e4def7afd9419ea98dfbf299777406 P d6d8a4e8ae7e492ab63f7ea9d8f076f3 [term 1 FOLLOWER]: Refusing update from remote peer 2e85b04a255a4c8698f45285e2969121: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:51:56.452868 5330 raft_consensus.cc:1261] T e1e4def7afd9419ea98dfbf299777406 P 1027d5f8e9774183b38a370e3312a2e3 [term 1 FOLLOWER]: Refusing update from remote peer 2e85b04a255a4c8698f45285e2969121: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:51:56.453763 5534 consensus_queue.cc:1035] T e1e4def7afd9419ea98dfbf299777406 P 2e85b04a255a4c8698f45285e2969121 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1027d5f8e9774183b38a370e3312a2e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:51:56.454142 5529 consensus_queue.cc:1035] T e1e4def7afd9419ea98dfbf299777406 P 2e85b04a255a4c8698f45285e2969121 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6d8a4e8ae7e492ab63f7ea9d8f076f3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:51:56.465574 5539 mvcc.cc:205] Tried to move back new op lower bound from 7334695388982026240 to 7334695388836593664. Current Snapshot: MvccSnapshot[applied={T|T < 7334695388982026240}]
I20260929 15:51:56.477249 5058 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:51:56.477548 5058 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:51:56.480945 5058 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ab8cf92e214431e8ef4b052a60bb1aa: Sending DeleteTablet for 3 replicas of tablet e1e4def7afd9419ea98dfbf299777406
I20260929 15:51:56.482072 5180 tablet_service.cc:1514] Processing DeleteTablet for tablet e1e4def7afd9419ea98dfbf299777406 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:51:56 UTC) from {username='slave'} at 127.0.0.1:57142
I20260929 15:51:56.482076 5440 tablet_service.cc:1514] Processing DeleteTablet for tablet e1e4def7afd9419ea98dfbf299777406 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:51:56 UTC) from {username='slave'} at 127.0.0.1:46710
I20260929 15:51:56.482393 5310 tablet_service.cc:1514] Processing DeleteTablet for tablet e1e4def7afd9419ea98dfbf299777406 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:51:56 UTC) from {username='slave'} at 127.0.0.1:36710
I20260929 15:51:56.483958 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 5118
I20260929 15:51:56.492362 5548 tablet_replica.cc:330] T e1e4def7afd9419ea98dfbf299777406 P 2e85b04a255a4c8698f45285e2969121: stopping tablet replica
I20260929 15:51:56.492851 5547 tablet_replica.cc:330] T e1e4def7afd9419ea98dfbf299777406 P d6d8a4e8ae7e492ab63f7ea9d8f076f3: stopping tablet replica
I20260929 15:51:56.493323 5547 raft_consensus.cc:2229] T e1e4def7afd9419ea98dfbf299777406 P d6d8a4e8ae7e492ab63f7ea9d8f076f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:51:56.493793 5548 raft_consensus.cc:2229] T e1e4def7afd9419ea98dfbf299777406 P 2e85b04a255a4c8698f45285e2969121 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:51:56.493883 5547 raft_consensus.cc:2258] T e1e4def7afd9419ea98dfbf299777406 P d6d8a4e8ae7e492ab63f7ea9d8f076f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:51:56.495085 5519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:56.485146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57156 (local address 127.0.97.1:44729)
0929 15:51:56.485374 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:56.485381 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:56.492033 (+ 6652us) server_negotiation.cc:407] Connection header received
0929 15:51:56.492413 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:56.492420 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:56.492491 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:56.492577 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:56.494309 (+ 1732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:56.494314 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:56.494319 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:51:56.494344 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:56.494370 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:51:56.494382 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:56.494464 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:56.494738 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 15:51:56.494797 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 15:51:56.495931 5547 ts_tablet_manager.cc:1905] T e1e4def7afd9419ea98dfbf299777406 P d6d8a4e8ae7e492ab63f7ea9d8f076f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:51:56.496251 5240 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:51:56.497063 5549 tablet_replica.cc:330] T e1e4def7afd9419ea98dfbf299777406 P 1027d5f8e9774183b38a370e3312a2e3: stopping tablet replica
I20260929 15:51:56.497476 5549 raft_consensus.cc:2229] T e1e4def7afd9419ea98dfbf299777406 P 1027d5f8e9774183b38a370e3312a2e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:51:56.498061 5549 raft_consensus.cc:2258] T e1e4def7afd9419ea98dfbf299777406 P 1027d5f8e9774183b38a370e3312a2e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:51:56.500645 5549 ts_tablet_manager.cc:1905] T e1e4def7afd9419ea98dfbf299777406 P 1027d5f8e9774183b38a370e3312a2e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:51:56.502839 5547 ts_tablet_manager.cc:1918] T e1e4def7afd9419ea98dfbf299777406 P d6d8a4e8ae7e492ab63f7ea9d8f076f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:51:56.503010 5547 log.cc:1196] T e1e4def7afd9419ea98dfbf299777406 P d6d8a4e8ae7e492ab63f7ea9d8f076f3: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-2/wal/wals/e1e4def7afd9419ea98dfbf299777406
I20260929 15:51:56.503453 5547 ts_tablet_manager.cc:1939] T e1e4def7afd9419ea98dfbf299777406 P d6d8a4e8ae7e492ab63f7ea9d8f076f3: Deleting consensus metadata
I20260929 15:51:56.504878 5046 catalog_manager.cc:4841] TS d6d8a4e8ae7e492ab63f7ea9d8f076f3 (127.0.97.3:42867): tablet e1e4def7afd9419ea98dfbf299777406 (table test-table [id=0deb3a2d5b8740b69a8408ef5850726f]) successfully deleted
I20260929 15:51:56.507581 5549 ts_tablet_manager.cc:1918] T e1e4def7afd9419ea98dfbf299777406 P 1027d5f8e9774183b38a370e3312a2e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:51:56.507747 5549 log.cc:1196] T e1e4def7afd9419ea98dfbf299777406 P 1027d5f8e9774183b38a370e3312a2e3: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697075657932-388-0/minicluster-data/ts-1/wal/wals/e1e4def7afd9419ea98dfbf299777406
I20260929 15:51:56.508301 5549 ts_tablet_manager.cc:1939] T e1e4def7afd9419ea98dfbf299777406 P 1027d5f8e9774183b38a370e3312a2e3: Deleting consensus metadata
I20260929 15:51:56.510141 5044 catalog_manager.cc:4841] TS 1027d5f8e9774183b38a370e3312a2e3 (127.0.97.2:34589): tablet e1e4def7afd9419ea98dfbf299777406 (table test-table [id=0deb3a2d5b8740b69a8408ef5850726f]) successfully deleted
I20260929 15:51:56.588869 5548 raft_consensus.cc:2258] T e1e4def7afd9419ea98dfbf299777406 P 2e85b04a255a4c8698f45285e2969121 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:51:56.590144 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 5118
I20260929 15:51:56.607682 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 5248
W20260929 15:51:56.607658 5045 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:44729 (error 108)
I20260929 15:51:56.609597 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:56.609099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44729 (local address 127.0.0.1:57166)
0929 15:51:56.609254 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:56.609406 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 15:51:56.609786 5045 catalog_manager.cc:4584] TS 2e85b04a255a4c8698f45285e2969121 (127.0.97.1:44729): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e1e4def7afd9419ea98dfbf299777406: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44729: connect: Connection refused (error 111)
I20260929 15:51:56.610767 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:56.608520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36722 (local address 127.0.97.2:34589)
0929 15:51:56.608702 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:56.608709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:56.608774 (+ 65us) server_negotiation.cc:407] Connection header received
0929 15:51:56.609014 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:56.609019 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:56.609098 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:56.609210 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:56.610163 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:56.610168 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:56.610174 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:51:56.610198 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:56.610226 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:51:56.610237 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:56.610367 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:56.610455 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 15:51:56.610511 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 15:51:56.611356 5370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:51:56.663094 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:56.662662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44729 (local address 127.0.0.1:57176)
0929 15:51:56.662846 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:56.662954 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 15:51:56.664077 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:56.663783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44729 (local address 127.0.0.1:57190)
0929 15:51:56.663907 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:56.663971 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 15:51:56.696635 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 5248
I20260929 15:51:56.712989 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 5378
I20260929 15:51:56.715837 5520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:56.713810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46718 (local address 127.0.97.3:42867)
0929 15:51:56.714024 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:56.714032 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:56.714055 (+ 23us) server_negotiation.cc:407] Connection header received
0929 15:51:56.714123 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:56.714126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:56.714210 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:56.714319 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:56.715214 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:56.715220 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:56.715225 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:51:56.715241 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:56.715278 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:51:56.715290 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:56.715418 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:56.715506 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 15:51:56.715562 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 15:51:56.716434 5500 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:51:56.724385 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:56.723980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44729 (local address 127.0.0.1:57194)
0929 15:51:56.724170 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:56.724249 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 15:51:56.725232 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:56.724927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44729 (local address 127.0.0.1:57198)
0929 15:51:56.725062 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:56.725130 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 15:51:56.801600 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 5378
I20260929 15:51:56.805683 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:56.805077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44729 (local address 127.0.0.1:57212)
0929 15:51:56.805334 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:56.805477 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 15:51:56.806838 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:56.806355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44729 (local address 127.0.0.1:57218)
0929 15:51:56.806566 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:56.806657 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260929 15:51:56.818358 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 5028
I20260929 15:51:56.819500 5089 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:51:56.923709 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 5028
2026-09-29T15:51:56Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2653 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 15:51:56.966316 388 mini_oidc.cc:152] Starting JWKS server
I20260929 15:51:56.968705 388 webserver.cc:460] Webserver started at https://127.0.0.1:38747/ using document root <none> and password file <none>
I20260929 15:51:56.969003 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 15:51:56.969630 388 webserver.cc:460] Webserver started at http://127.0.0.1:45547/ using document root <none> and password file <none>
2026-09-29T15:51:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:51:56Z Disabled control of system clock
I20260929 15:51:56.994534 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43931
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:38215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43931
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38747/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697075657932-388-0/testchainca.cert with env {}
W20260929 15:51:57.304589 5561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:51:57.304868 5561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:51:57.304953 5561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:51:57.312167 5561 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:51:57.312268 5561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:51:57.312305 5561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:51:57.312358 5561 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:51:57.312408 5561 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 15:51:57.312462 5561 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 15:51:57.312502 5561 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38747/jwks.json
I20260929 15:51:57.322238 5561 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43931
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38747/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697075657932-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:51:57.323403 5561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:51:57.324585 5561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:51:57.336229 5569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:57.336185 5567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:57.336259 5566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:51:57.337100 5561 server_base.cc:1034] running on GCE node
I20260929 15:51:57.337863 5561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:51:57.339078 5561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:51:57.340307 5561 hybrid_clock.cc:648] HybridClock initialized: now 1790697117340272 us; error 50 us; skew 500 ppm
I20260929 15:51:57.340655 5561 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:51:57.342942 5561 webserver.cc:460] Webserver started at http://127.0.97.62:39607/ using document root <none> and password file <none>
I20260929 15:51:57.343597 5561 fs_manager.cc:359] Metadata directory not provided
I20260929 15:51:57.343724 5561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:51:57.344031 5561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:51:57.346845 5561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "992fbb1351444651823035ddfccbee32"
format_stamp: "Formatted at 2026-09-29 15:51:57 on dist-test-slave-r0ck"
server_key: "3e4a6b52b8b430379688c87171241615"
server_key_iv: "ddc8aa0a4ab173a55aafd2d27f6ca56d"
server_key_version: "encryptionkey@0"
I20260929 15:51:57.347612 5561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "992fbb1351444651823035ddfccbee32"
format_stamp: "Formatted at 2026-09-29 15:51:57 on dist-test-slave-r0ck"
server_key: "3e4a6b52b8b430379688c87171241615"
server_key_iv: "ddc8aa0a4ab173a55aafd2d27f6ca56d"
server_key_version: "encryptionkey@0"
I20260929 15:51:57.352450 5561 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 15:51:57.355721 5575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:57.357074 5561 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 15:51:57.357252 5561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "992fbb1351444651823035ddfccbee32"
format_stamp: "Formatted at 2026-09-29 15:51:57 on dist-test-slave-r0ck"
server_key: "3e4a6b52b8b430379688c87171241615"
server_key_iv: "ddc8aa0a4ab173a55aafd2d27f6ca56d"
server_key_version: "encryptionkey@0"
I20260929 15:51:57.357498 5561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:51:57.372376 5561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:51:57.373109 5561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:51:57.373412 5561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:51:57.391403 5561 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43931
I20260929 15:51:57.391426 5626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43931 every 8 connection(s)
I20260929 15:51:57.392644 5561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:51:57.396067 5627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:57.400846 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 5561
I20260929 15:51:57.401046 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:51:57.401762 388 external_mini_cluster.cc:1428] Setting key 14604178929e1a1dbca2e25b5b0e3c3f
I20260929 15:51:57.406116 5627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 992fbb1351444651823035ddfccbee32: Bootstrap starting.
I20260929 15:51:57.409294 5627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 992fbb1351444651823035ddfccbee32: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:57.410508 5627 log.cc:824] T 00000000000000000000000000000000 P 992fbb1351444651823035ddfccbee32: Log is configured to *not* fsync() on all Append() calls
I20260929 15:51:57.413609 5627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 992fbb1351444651823035ddfccbee32: No bootstrap required, opened a new log
I20260929 15:51:57.415912 5630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:57.403591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54960 (local address 127.0.97.62:43931)
0929 15:51:57.404584 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:57.404593 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:57.404632 (+ 39us) server_negotiation.cc:407] Connection header received
0929 15:51:57.406279 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:57.406314 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:57.406706 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:57.406989 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:57.407637 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:57.408901 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:57.410858 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:57.411919 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:57.412056 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:57.413307 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:57.413316 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:57.413330 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:51:57.414123 (+ 793us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:57.414200 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:51:57.414221 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:57.414341 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:57.414589 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 15:51:57.414922 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":383,"threads_started":1}
I20260929 15:51:57.420403 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:51:57.423859 5627 raft_consensus.cc:348] T 00000000000000000000000000000000 P 992fbb1351444651823035ddfccbee32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "992fbb1351444651823035ddfccbee32" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43931 } }
I20260929 15:51:57.424276 5627 raft_consensus.cc:374] T 00000000000000000000000000000000 P 992fbb1351444651823035ddfccbee32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:57.424387 5627 raft_consensus.cc:729] T 00000000000000000000000000000000 P 992fbb1351444651823035ddfccbee32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 992fbb1351444651823035ddfccbee32, State: Initialized, Role: FOLLOWER
I20260929 15:51:57.424932 5627 consensus_queue.cc:260] T 00000000000000000000000000000000 P 992fbb1351444651823035ddfccbee32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "992fbb1351444651823035ddfccbee32" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43931 } }
I20260929 15:51:57.425122 5627 raft_consensus.cc:388] T 00000000000000000000000000000000 P 992fbb1351444651823035ddfccbee32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:51:57.425220 5627 raft_consensus.cc:482] T 00000000000000000000000000000000 P 992fbb1351444651823035ddfccbee32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:51:57.425421 5627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 992fbb1351444651823035ddfccbee32 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:57.427909 5627 raft_consensus.cc:504] T 00000000000000000000000000000000 P 992fbb1351444651823035ddfccbee32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "992fbb1351444651823035ddfccbee32" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43931 } }
I20260929 15:51:57.428321 5627 leader_election.cc:302] T 00000000000000000000000000000000 P 992fbb1351444651823035ddfccbee32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 992fbb1351444651823035ddfccbee32; no voters:
I20260929 15:51:57.428882 5627 leader_election.cc:288] T 00000000000000000000000000000000 P 992fbb1351444651823035ddfccbee32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:51:57.429037 5632 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 992fbb1351444651823035ddfccbee32 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:51:57.429754 5632 raft_consensus.cc:686] T 00000000000000000000000000000000 P 992fbb1351444651823035ddfccbee32 [term 1 LEADER]: Becoming Leader. State: Replica: 992fbb1351444651823035ddfccbee32, State: Running, Role: LEADER
I20260929 15:51:57.430280 5632 consensus_queue.cc:237] T 00000000000000000000000000000000 P 992fbb1351444651823035ddfccbee32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "992fbb1351444651823035ddfccbee32" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43931 } }
I20260929 15:51:57.430881 5627 sys_catalog.cc:564] T 00000000000000000000000000000000 P 992fbb1351444651823035ddfccbee32 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:51:57.433789 5633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 992fbb1351444651823035ddfccbee32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "992fbb1351444651823035ddfccbee32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "992fbb1351444651823035ddfccbee32" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43931 } } }
I20260929 15:51:57.434134 5633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 992fbb1351444651823035ddfccbee32 [sys.catalog]: This master's current role is: LEADER
I20260929 15:51:57.436376 5634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 992fbb1351444651823035ddfccbee32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 992fbb1351444651823035ddfccbee32. Latest consensus state: current_term: 1 leader_uuid: "992fbb1351444651823035ddfccbee32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "992fbb1351444651823035ddfccbee32" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43931 } } }
I20260929 15:51:57.436798 5634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 992fbb1351444651823035ddfccbee32 [sys.catalog]: This master's current role is: LEADER
I20260929 15:51:57.437925 5640 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:51:57.442839 5640 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:51:57.451390 5640 catalog_manager.cc:1347] Generated new cluster ID: 7be42c14996742919506eb96bf3185e1
I20260929 15:51:57.451515 5640 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:51:57.487139 5640 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:51:57.488381 5640 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:51:57.500837 5640 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 992fbb1351444651823035ddfccbee32: Generated new TSK 0
I20260929 15:51:57.501715 5640 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 15:51:57.537643 5657 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 15:51:57.537897 5657 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 15:51:57.538877 5630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:57.516990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54964 (local address 127.0.97.62:43931)
0929 15:51:57.517182 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:57.517189 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:57.517790 (+ 601us) server_negotiation.cc:407] Connection header received
0929 15:51:57.518038 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:57.518044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:57.518141 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:57.518290 (+ 149us) server_negotiation.cc:226] Negotiated authn=JWT
0929 15:51:57.518786 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:57.519902 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:57.521258 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:57.522045 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:57.522178 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:57.522662 (+ 484us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 15:51:57.538263 (+ 15601us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38747/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 15:51:57.538601 (+ 338us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54964: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38747/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":61}
W20260929 15:51:57.538998 5630 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54964: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38747/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 15:51:57.540230 5656 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:51:57.516987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43931 (local address 127.0.0.1:54964)
0929 15:51:57.517700 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:57.517715 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:57.517851 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:57.518330 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:57.518333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:57.518343 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0929 15:51:57.518655 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:57.518662 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:57.520035 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:57.520038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:57.521125 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:57.521135 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:57.522167 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:57.522226 (+ 59us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:57.522365 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:57.522420 (+ 55us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 15:51:57.538532 (+ 16112us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38747/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 15:51:57.538815 (+ 283us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:43931: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38747/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":414,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:43931: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38747/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 15:51:57.542026 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 5561
I20260929 15:51:57.543084 5622 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:51:57.635954 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 5561
2026-09-29T15:51:57Z chronyd exiting
I20260929 15:51:57.685961 388 test_util.cc:175] -----------------------------------------------
I20260929 15:51:57.686125 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697075657932-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (723 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 15:51:57.689772 388 mini_oidc.cc:152] Starting JWKS server
I20260929 15:51:57.691591 388 webserver.cc:460] Webserver started at https://127.0.0.1:35527/ using document root <none> and password file <none>
I20260929 15:51:57.691820 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 15:51:57.692484 388 webserver.cc:460] Webserver started at http://127.0.0.1:44303/ using document root <none> and password file <none>
2026-09-29T15:51:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:51:57Z Disabled control of system clock
I20260929 15:51:57.718215 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44501
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44501
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35527/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697075657932-388-0/testchainca.cert with env {}
W20260929 15:51:58.029631 5664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:51:58.029907 5664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:51:58.029994 5664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:51:58.037132 5664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:51:58.037245 5664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:51:58.037281 5664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:51:58.037381 5664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:51:58.037452 5664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 15:51:58.037509 5664 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 15:51:58.037552 5664 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35527/jwks.json
I20260929 15:51:58.047224 5664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44501
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35527/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697075657932-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:51:58.048429 5664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:51:58.049682 5664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:51:58.061240 5669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:58.061218 5670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:58.061524 5672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:51:58.062083 5664 server_base.cc:1034] running on GCE node
I20260929 15:51:58.062906 5664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:51:58.064378 5664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:51:58.065627 5664 hybrid_clock.cc:648] HybridClock initialized: now 1790697118065577 us; error 44 us; skew 500 ppm
I20260929 15:51:58.066030 5664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:51:58.068297 5664 webserver.cc:460] Webserver started at http://127.0.97.62:41855/ using document root <none> and password file <none>
I20260929 15:51:58.068985 5664 fs_manager.cc:359] Metadata directory not provided
I20260929 15:51:58.069118 5664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:51:58.069500 5664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:51:58.073894 5664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "65a5caa598584424a64945320d5de595"
format_stamp: "Formatted at 2026-09-29 15:51:58 on dist-test-slave-r0ck"
server_key: "054c68d1c182cad5a97c3cecc7c50378"
server_key_iv: "7ebe181fd42b74845d55630a77ec69a4"
server_key_version: "encryptionkey@0"
I20260929 15:51:58.074680 5664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "65a5caa598584424a64945320d5de595"
format_stamp: "Formatted at 2026-09-29 15:51:58 on dist-test-slave-r0ck"
server_key: "054c68d1c182cad5a97c3cecc7c50378"
server_key_iv: "7ebe181fd42b74845d55630a77ec69a4"
server_key_version: "encryptionkey@0"
I20260929 15:51:58.079969 5664 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 15:51:58.083461 5678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:58.084786 5664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 15:51:58.084998 5664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "65a5caa598584424a64945320d5de595"
format_stamp: "Formatted at 2026-09-29 15:51:58 on dist-test-slave-r0ck"
server_key: "054c68d1c182cad5a97c3cecc7c50378"
server_key_iv: "7ebe181fd42b74845d55630a77ec69a4"
server_key_version: "encryptionkey@0"
I20260929 15:51:58.085213 5664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:51:58.117610 5664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:51:58.118428 5664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:51:58.118709 5664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:51:58.136724 5664 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44501
I20260929 15:51:58.136759 5729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44501 every 8 connection(s)
I20260929 15:51:58.138104 5664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:51:58.138639 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 5664
I20260929 15:51:58.138839 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:51:58.139657 388 external_mini_cluster.cc:1428] Setting key 2f6642fbeba8e0ff835616c6edef2952
I20260929 15:51:58.142323 5730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:58.152467 5733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:58.141520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56076 (local address 127.0.97.62:44501)
0929 15:51:58.142590 (+ 1070us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:58.142609 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:58.142650 (+ 41us) server_negotiation.cc:407] Connection header received
0929 15:51:58.144211 (+ 1561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:58.144230 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:58.144667 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:58.145085 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:58.145642 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:58.146954 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:58.148232 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:58.148870 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:58.149065 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:58.150265 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:58.150272 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:58.150284 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:51:58.150835 (+ 551us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:58.150905 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:51:58.150924 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:58.151052 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:58.151266 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 15:51:58.151555 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":390,"threads_started":1}
I20260929 15:51:58.152606 5730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65a5caa598584424a64945320d5de595: Bootstrap starting.
I20260929 15:51:58.156301 5730 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65a5caa598584424a64945320d5de595: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:58.156344 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:51:58.157245 5730 log.cc:824] T 00000000000000000000000000000000 P 65a5caa598584424a64945320d5de595: Log is configured to *not* fsync() on all Append() calls
I20260929 15:51:58.160315 5730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65a5caa598584424a64945320d5de595: No bootstrap required, opened a new log
I20260929 15:51:58.167766 5730 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65a5caa598584424a64945320d5de595 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a5caa598584424a64945320d5de595" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44501 } }
I20260929 15:51:58.168109 5730 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65a5caa598584424a64945320d5de595 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:58.168227 5730 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65a5caa598584424a64945320d5de595 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65a5caa598584424a64945320d5de595, State: Initialized, Role: FOLLOWER
I20260929 15:51:58.168778 5730 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65a5caa598584424a64945320d5de595 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a5caa598584424a64945320d5de595" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44501 } }
I20260929 15:51:58.168969 5730 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65a5caa598584424a64945320d5de595 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:51:58.169111 5730 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65a5caa598584424a64945320d5de595 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:51:58.169274 5730 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65a5caa598584424a64945320d5de595 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:58.171327 5730 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65a5caa598584424a64945320d5de595 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a5caa598584424a64945320d5de595" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44501 } }
I20260929 15:51:58.171830 5730 leader_election.cc:302] T 00000000000000000000000000000000 P 65a5caa598584424a64945320d5de595 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65a5caa598584424a64945320d5de595; no voters:
I20260929 15:51:58.172431 5730 leader_election.cc:288] T 00000000000000000000000000000000 P 65a5caa598584424a64945320d5de595 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:51:58.172628 5735 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65a5caa598584424a64945320d5de595 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:51:58.173596 5735 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65a5caa598584424a64945320d5de595 [term 1 LEADER]: Becoming Leader. State: Replica: 65a5caa598584424a64945320d5de595, State: Running, Role: LEADER
I20260929 15:51:58.174108 5735 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65a5caa598584424a64945320d5de595 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a5caa598584424a64945320d5de595" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44501 } }
I20260929 15:51:58.174666 5730 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65a5caa598584424a64945320d5de595 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:51:58.176903 5736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65a5caa598584424a64945320d5de595 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65a5caa598584424a64945320d5de595" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a5caa598584424a64945320d5de595" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44501 } } }
I20260929 15:51:58.177525 5736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65a5caa598584424a64945320d5de595 [sys.catalog]: This master's current role is: LEADER
I20260929 15:51:58.177364 5737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65a5caa598584424a64945320d5de595 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65a5caa598584424a64945320d5de595. Latest consensus state: current_term: 1 leader_uuid: "65a5caa598584424a64945320d5de595" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a5caa598584424a64945320d5de595" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44501 } } }
I20260929 15:51:58.177735 5737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65a5caa598584424a64945320d5de595 [sys.catalog]: This master's current role is: LEADER
I20260929 15:51:58.179203 5743 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:51:58.183925 5743 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:51:58.193889 5743 catalog_manager.cc:1347] Generated new cluster ID: f1600f7a9f3f4da3af61bc384dc9aef5
I20260929 15:51:58.194008 5743 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:51:58.216373 5743 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:51:58.218214 5743 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:51:58.227944 5743 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65a5caa598584424a64945320d5de595: Generated new TSK 0
I20260929 15:51:58.228950 5743 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 15:51:58.266322 5760 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 15:51:58.266448 5760 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 15:51:58.267474 5733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:58.247709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56080 (local address 127.0.97.62:44501)
0929 15:51:58.247933 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:58.247942 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:58.248439 (+ 497us) server_negotiation.cc:407] Connection header received
0929 15:51:58.248641 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:58.248645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:58.248712 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:58.248845 (+ 133us) server_negotiation.cc:226] Negotiated authn=JWT
0929 15:51:58.249442 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:58.250140 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:58.252091 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:58.252552 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:58.252643 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:58.253207 (+ 564us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 15:51:58.266931 (+ 13724us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35527/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 15:51:58.267238 (+ 307us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56080: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35527/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":63}
W20260929 15:51:58.267614 5733 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56080: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35527/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260929 15:51:58.267884 5759 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:51:58.247681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44501 (local address 127.0.0.1:56080)
0929 15:51:58.248324 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:58.248353 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:58.248499 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:58.248892 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:58.248897 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:58.248910 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0929 15:51:58.249224 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:58.249237 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:58.250270 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:58.250276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:58.251948 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:58.251962 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:58.252669 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:58.252715 (+ 46us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:58.252921 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:58.252963 (+ 42us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 15:51:58.267178 (+ 14215us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35527/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 15:51:58.267489 (+ 311us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:44501: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35527/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":356,"threads_started":1}
I20260929 15:51:58.268905 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 5664
I20260929 15:51:58.270180 5725 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:51:58.360625 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 5664
2026-09-29T15:51:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (725 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 15:51:58.415393 388 mini_oidc.cc:152] Starting JWKS server
I20260929 15:51:58.417207 388 webserver.cc:460] Webserver started at https://127.0.0.1:36687/ using document root <none> and password file <none>
I20260929 15:51:58.417500 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 15:51:58.418107 388 webserver.cc:460] Webserver started at http://127.0.0.1:39547/ using document root <none> and password file <none>
2026-09-29T15:51:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:51:58Z Disabled control of system clock
I20260929 15:51:58.444757 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39403
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:46281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39403
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36687/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 15:51:58.753681 5767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:51:58.753943 5767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:51:58.754024 5767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:51:58.760965 5767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:51:58.761059 5767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:51:58.761096 5767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:51:58.761150 5767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:51:58.761193 5767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 15:51:58.761245 5767 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 15:51:58.761283 5767 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36687/jwks.json
I20260929 15:51:58.770198 5767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39403
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36687/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:51:58.771304 5767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:51:58.772928 5767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:51:58.783885 5772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:58.783941 5775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:58.783951 5773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:51:58.784972 5767 server_base.cc:1034] running on GCE node
I20260929 15:51:58.785710 5767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:51:58.786962 5767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:51:58.788157 5767 hybrid_clock.cc:648] HybridClock initialized: now 1790697118788120 us; error 34 us; skew 500 ppm
I20260929 15:51:58.788482 5767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:51:58.790722 5767 webserver.cc:460] Webserver started at http://127.0.97.62:40875/ using document root <none> and password file <none>
I20260929 15:51:58.791350 5767 fs_manager.cc:359] Metadata directory not provided
I20260929 15:51:58.791466 5767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:51:58.791791 5767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:51:58.794524 5767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "1432201c48e3405ca86151e21c44e042"
format_stamp: "Formatted at 2026-09-29 15:51:58 on dist-test-slave-r0ck"
server_key: "12f050ea4fbff196031a5bd52ea8fffc"
server_key_iv: "8a0e41977e686d985ab60d3cf9fedd4d"
server_key_version: "encryptionkey@0"
I20260929 15:51:58.795483 5767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "1432201c48e3405ca86151e21c44e042"
format_stamp: "Formatted at 2026-09-29 15:51:58 on dist-test-slave-r0ck"
server_key: "12f050ea4fbff196031a5bd52ea8fffc"
server_key_iv: "8a0e41977e686d985ab60d3cf9fedd4d"
server_key_version: "encryptionkey@0"
I20260929 15:51:58.800149 5767 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260929 15:51:58.803352 5781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:58.804517 5767 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 15:51:58.804699 5767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "1432201c48e3405ca86151e21c44e042"
format_stamp: "Formatted at 2026-09-29 15:51:58 on dist-test-slave-r0ck"
server_key: "12f050ea4fbff196031a5bd52ea8fffc"
server_key_iv: "8a0e41977e686d985ab60d3cf9fedd4d"
server_key_version: "encryptionkey@0"
I20260929 15:51:58.804908 5767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:51:58.819449 5767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:51:58.820163 5767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:51:58.820477 5767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:51:58.838011 5767 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39403
I20260929 15:51:58.838030 5832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39403 every 8 connection(s)
I20260929 15:51:58.839227 5767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:51:58.841063 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 5767
I20260929 15:51:58.841269 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:51:58.842108 388 external_mini_cluster.cc:1428] Setting key 38da7ac06595dbbc293071ff0482d5d6
I20260929 15:51:58.843586 5833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:58.855144 5836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:58.843896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36680 (local address 127.0.97.62:39403)
0929 15:51:58.844798 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:58.844808 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:58.844846 (+ 38us) server_negotiation.cc:407] Connection header received
0929 15:51:58.846335 (+ 1489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:58.846356 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:58.846779 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:58.847093 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:58.847656 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:58.848987 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:58.850821 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:58.851454 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:58.851580 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:58.852828 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:58.852836 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:58.852853 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:51:58.853462 (+ 609us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:58.853546 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:51:58.853569 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:58.853658 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:58.853883 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 15:51:58.854224 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":343,"threads_started":1}
I20260929 15:51:58.856220 5833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1432201c48e3405ca86151e21c44e042: Bootstrap starting.
I20260929 15:51:58.858685 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:51:58.859535 5833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1432201c48e3405ca86151e21c44e042: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:58.860697 5833 log.cc:824] T 00000000000000000000000000000000 P 1432201c48e3405ca86151e21c44e042: Log is configured to *not* fsync() on all Append() calls
I20260929 15:51:58.863454 5833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1432201c48e3405ca86151e21c44e042: No bootstrap required, opened a new log
I20260929 15:51:58.870343 5833 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1432201c48e3405ca86151e21c44e042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1432201c48e3405ca86151e21c44e042" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39403 } }
I20260929 15:51:58.870641 5833 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1432201c48e3405ca86151e21c44e042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:58.870708 5833 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1432201c48e3405ca86151e21c44e042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1432201c48e3405ca86151e21c44e042, State: Initialized, Role: FOLLOWER
I20260929 15:51:58.871150 5833 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1432201c48e3405ca86151e21c44e042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1432201c48e3405ca86151e21c44e042" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39403 } }
I20260929 15:51:58.871331 5833 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1432201c48e3405ca86151e21c44e042 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:51:58.871420 5833 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1432201c48e3405ca86151e21c44e042 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:51:58.871536 5833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1432201c48e3405ca86151e21c44e042 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:58.873553 5833 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1432201c48e3405ca86151e21c44e042 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1432201c48e3405ca86151e21c44e042" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39403 } }
I20260929 15:51:58.873968 5833 leader_election.cc:302] T 00000000000000000000000000000000 P 1432201c48e3405ca86151e21c44e042 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1432201c48e3405ca86151e21c44e042; no voters:
I20260929 15:51:58.874460 5833 leader_election.cc:288] T 00000000000000000000000000000000 P 1432201c48e3405ca86151e21c44e042 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:51:58.874612 5838 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1432201c48e3405ca86151e21c44e042 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:51:58.875880 5838 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1432201c48e3405ca86151e21c44e042 [term 1 LEADER]: Becoming Leader. State: Replica: 1432201c48e3405ca86151e21c44e042, State: Running, Role: LEADER
I20260929 15:51:58.876338 5838 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1432201c48e3405ca86151e21c44e042 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1432201c48e3405ca86151e21c44e042" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39403 } }
I20260929 15:51:58.876840 5833 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1432201c48e3405ca86151e21c44e042 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:51:58.879261 5840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1432201c48e3405ca86151e21c44e042 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1432201c48e3405ca86151e21c44e042. Latest consensus state: current_term: 1 leader_uuid: "1432201c48e3405ca86151e21c44e042" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1432201c48e3405ca86151e21c44e042" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39403 } } }
I20260929 15:51:58.879746 5840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1432201c48e3405ca86151e21c44e042 [sys.catalog]: This master's current role is: LEADER
I20260929 15:51:58.879287 5839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1432201c48e3405ca86151e21c44e042 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1432201c48e3405ca86151e21c44e042" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1432201c48e3405ca86151e21c44e042" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39403 } } }
I20260929 15:51:58.880535 5839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1432201c48e3405ca86151e21c44e042 [sys.catalog]: This master's current role is: LEADER
I20260929 15:51:58.881824 5845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:51:58.886739 5845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:51:58.895681 5845 catalog_manager.cc:1347] Generated new cluster ID: 23a863d554ef406cb833b64973025a82
I20260929 15:51:58.895802 5845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:51:58.908645 5845 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:51:58.910076 5845 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:51:58.916241 5845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1432201c48e3405ca86151e21c44e042: Generated new TSK 0
I20260929 15:51:58.916972 5845 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:51:58.939941 5836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:58.932493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36684 (local address 127.0.97.62:39403)
0929 15:51:58.932638 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:58.932642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:58.933029 (+ 387us) server_negotiation.cc:407] Connection header received
0929 15:51:58.933293 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:58.933297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:58.933416 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:58.933583 (+ 167us) server_negotiation.cc:226] Negotiated authn=JWT
0929 15:51:58.934044 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:58.934742 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:58.936065 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:58.936512 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:58.936646 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:58.937118 (+ 472us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 15:51:58.939339 (+ 2221us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36687/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 15:51:58.939641 (+ 302us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36684: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36687/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":40}
W20260929 15:51:58.940057 5836 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36684: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36687/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 15:51:58.942106 5862 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:51:58.932339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39403 (local address 127.0.0.1:36684)
0929 15:51:58.932926 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:58.932941 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:58.933099 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:58.933597 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:58.933602 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:58.933614 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0929 15:51:58.933903 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:58.933912 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:58.934882 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:58.934889 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:58.935922 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:58.935932 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:58.936636 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:58.936707 (+ 71us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:58.936848 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:58.936878 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 15:51:58.939545 (+ 2667us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36687/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 15:51:58.941658 (+ 2113us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:39403: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36687/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":1827,"thread_start_us":328,"threads_started":1}
I20260929 15:51:58.943528 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 5767
I20260929 15:51:58.944676 5828 generic_service.cc:196] Checking for leaks (request via RPC)
E20260929 15:51:58.955117 5863 webserver.cc:533] Webserver: sslize failed:
E20260929 15:51:58.955243 5863 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 15:51:59.050894 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 5767
2026-09-29T15:51:59Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (690 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T15:51:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:51:59Z Disabled control of system clock
I20260929 15:51:59.130750 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35337
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:39851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35337
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697075657932-388-0/testchainca.cert with env {}
W20260929 15:51:59.444438 5868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:51:59.444742 5868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:51:59.444830 5868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:51:59.452106 5868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:51:59.452209 5868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:51:59.452247 5868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:51:59.452296 5868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:51:59.452340 5868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 15:51:59.452394 5868 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 15:51:59.452433 5868 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 15:51:59.462070 5868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35337
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697075657932-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:51:59.463265 5868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:51:59.464429 5868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:51:59.475843 5874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:59.475904 5876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:51:59.476177 5873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:51:59.476709 5868 server_base.cc:1034] running on GCE node
I20260929 15:51:59.477526 5868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:51:59.478863 5868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:51:59.480093 5868 hybrid_clock.cc:648] HybridClock initialized: now 1790697119480060 us; error 56 us; skew 500 ppm
I20260929 15:51:59.480423 5868 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:51:59.482578 5868 webserver.cc:460] Webserver started at http://127.0.97.62:40101/ using document root <none> and password file <none>
I20260929 15:51:59.483224 5868 fs_manager.cc:359] Metadata directory not provided
I20260929 15:51:59.483361 5868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:51:59.483664 5868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:51:59.486447 5868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "a46c79a5715f4c7bac867b0ae152f6c1"
format_stamp: "Formatted at 2026-09-29 15:51:59 on dist-test-slave-r0ck"
server_key: "eac19728e2125807b2705a246ac422c6"
server_key_iv: "2f73cf57f5dadbd4cb2c1902b34d1827"
server_key_version: "encryptionkey@0"
I20260929 15:51:59.487193 5868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "a46c79a5715f4c7bac867b0ae152f6c1"
format_stamp: "Formatted at 2026-09-29 15:51:59 on dist-test-slave-r0ck"
server_key: "eac19728e2125807b2705a246ac422c6"
server_key_iv: "2f73cf57f5dadbd4cb2c1902b34d1827"
server_key_version: "encryptionkey@0"
I20260929 15:51:59.491935 5868 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260929 15:51:59.495203 5882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:51:59.496450 5868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 15:51:59.496625 5868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "a46c79a5715f4c7bac867b0ae152f6c1"
format_stamp: "Formatted at 2026-09-29 15:51:59 on dist-test-slave-r0ck"
server_key: "eac19728e2125807b2705a246ac422c6"
server_key_iv: "2f73cf57f5dadbd4cb2c1902b34d1827"
server_key_version: "encryptionkey@0"
I20260929 15:51:59.496850 5868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:51:59.511231 5868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:51:59.512003 5868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:51:59.512277 5868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:51:59.530133 5868 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35337
I20260929 15:51:59.530153 5933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35337 every 8 connection(s)
I20260929 15:51:59.531414 5868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:51:59.534945 5934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:51:59.536769 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 5868
I20260929 15:51:59.536966 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:51:59.537660 388 external_mini_cluster.cc:1428] Setting key c0ebbd02c838722d985a700e40ee08ec
I20260929 15:51:59.545907 5934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a46c79a5715f4c7bac867b0ae152f6c1: Bootstrap starting.
I20260929 15:51:59.549315 5934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a46c79a5715f4c7bac867b0ae152f6c1: Neither blocks nor log segments found. Creating new log.
I20260929 15:51:59.550391 5934 log.cc:824] T 00000000000000000000000000000000 P a46c79a5715f4c7bac867b0ae152f6c1: Log is configured to *not* fsync() on all Append() calls
I20260929 15:51:59.550551 5937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:59.539482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57850 (local address 127.0.97.62:35337)
0929 15:51:59.540424 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:59.540434 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:59.540473 (+ 39us) server_negotiation.cc:407] Connection header received
0929 15:51:59.542083 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:59.542103 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:59.542537 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:59.542850 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:51:59.543403 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:59.544623 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:59.546410 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:59.547023 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:59.547148 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:59.548389 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:51:59.548396 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:51:59.548407 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:51:59.548930 (+ 523us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:51:59.549012 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:51:59.549031 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:51:59.549119 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 15:51:59.549431 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 15:51:59.549682 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":358,"threads_started":1}
I20260929 15:51:59.553107 5934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a46c79a5715f4c7bac867b0ae152f6c1: No bootstrap required, opened a new log
I20260929 15:51:59.553998 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:51:59.560499 5934 raft_consensus.cc:348] T 00000000000000000000000000000000 P a46c79a5715f4c7bac867b0ae152f6c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a46c79a5715f4c7bac867b0ae152f6c1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35337 } }
I20260929 15:51:59.560882 5934 raft_consensus.cc:374] T 00000000000000000000000000000000 P a46c79a5715f4c7bac867b0ae152f6c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:51:59.561000 5934 raft_consensus.cc:729] T 00000000000000000000000000000000 P a46c79a5715f4c7bac867b0ae152f6c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a46c79a5715f4c7bac867b0ae152f6c1, State: Initialized, Role: FOLLOWER
I20260929 15:51:59.561627 5934 consensus_queue.cc:260] T 00000000000000000000000000000000 P a46c79a5715f4c7bac867b0ae152f6c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a46c79a5715f4c7bac867b0ae152f6c1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35337 } }
I20260929 15:51:59.561859 5934 raft_consensus.cc:388] T 00000000000000000000000000000000 P a46c79a5715f4c7bac867b0ae152f6c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:51:59.561954 5934 raft_consensus.cc:482] T 00000000000000000000000000000000 P a46c79a5715f4c7bac867b0ae152f6c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:51:59.562067 5934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a46c79a5715f4c7bac867b0ae152f6c1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:51:59.564059 5934 raft_consensus.cc:504] T 00000000000000000000000000000000 P a46c79a5715f4c7bac867b0ae152f6c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a46c79a5715f4c7bac867b0ae152f6c1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35337 } }
I20260929 15:51:59.564505 5934 leader_election.cc:302] T 00000000000000000000000000000000 P a46c79a5715f4c7bac867b0ae152f6c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a46c79a5715f4c7bac867b0ae152f6c1; no voters:
I20260929 15:51:59.564982 5934 leader_election.cc:288] T 00000000000000000000000000000000 P a46c79a5715f4c7bac867b0ae152f6c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:51:59.565147 5939 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a46c79a5715f4c7bac867b0ae152f6c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:51:59.566262 5939 raft_consensus.cc:686] T 00000000000000000000000000000000 P a46c79a5715f4c7bac867b0ae152f6c1 [term 1 LEADER]: Becoming Leader. State: Replica: a46c79a5715f4c7bac867b0ae152f6c1, State: Running, Role: LEADER
I20260929 15:51:59.566900 5939 consensus_queue.cc:237] T 00000000000000000000000000000000 P a46c79a5715f4c7bac867b0ae152f6c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a46c79a5715f4c7bac867b0ae152f6c1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35337 } }
I20260929 15:51:59.567183 5934 sys_catalog.cc:564] T 00000000000000000000000000000000 P a46c79a5715f4c7bac867b0ae152f6c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:51:59.569900 5940 sys_catalog.cc:455] T 00000000000000000000000000000000 P a46c79a5715f4c7bac867b0ae152f6c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a46c79a5715f4c7bac867b0ae152f6c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a46c79a5715f4c7bac867b0ae152f6c1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35337 } } }
I20260929 15:51:59.570251 5940 sys_catalog.cc:458] T 00000000000000000000000000000000 P a46c79a5715f4c7bac867b0ae152f6c1 [sys.catalog]: This master's current role is: LEADER
I20260929 15:51:59.569860 5941 sys_catalog.cc:455] T 00000000000000000000000000000000 P a46c79a5715f4c7bac867b0ae152f6c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a46c79a5715f4c7bac867b0ae152f6c1. Latest consensus state: current_term: 1 leader_uuid: "a46c79a5715f4c7bac867b0ae152f6c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a46c79a5715f4c7bac867b0ae152f6c1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35337 } } }
I20260929 15:51:59.570842 5941 sys_catalog.cc:458] T 00000000000000000000000000000000 P a46c79a5715f4c7bac867b0ae152f6c1 [sys.catalog]: This master's current role is: LEADER
I20260929 15:51:59.572518 5947 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:51:59.577514 5947 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:51:59.585813 5947 catalog_manager.cc:1347] Generated new cluster ID: 962fc5f7c86544cd90b79def433c7b29
I20260929 15:51:59.585999 5947 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:51:59.602824 5947 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:51:59.604099 5947 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:51:59.614955 5947 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a46c79a5715f4c7bac867b0ae152f6c1: Generated new TSK 0
I20260929 15:51:59.615972 5947 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:52:00.076938 5937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:51:59.635110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57864 (local address 127.0.97.62:35337)
0929 15:51:59.635305 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 15:51:59.635313 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:51:59.635709 (+ 396us) server_negotiation.cc:407] Connection header received
0929 15:51:59.635906 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:51:59.635911 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:51:59.636009 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:51:59.636164 (+ 155us) server_negotiation.cc:226] Negotiated authn=JWT
0929 15:51:59.636569 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:59.637744 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:59.639066 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:51:59.639762 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:51:59.639875 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:59.640313 (+ 438us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 15:52:00.076380 (+436067us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 15:52:00.076713 (+ 333us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57864: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 15:52:00.077046 5937 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57864: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 15:52:00.077421 5963 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:51:59.634983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35337 (local address 127.0.0.1:57864)
0929 15:51:59.635617 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0929 15:51:59.635633 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:51:59.635745 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:51:59.636185 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:51:59.636189 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:51:59.636198 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0929 15:51:59.636428 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:59.636436 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:59.637890 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:59.637893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:59.638925 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:51:59.638937 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:51:59.639880 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:51:59.639921 (+ 41us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:51:59.640054 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:51:59.640083 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 15:52:00.076670 (+436587us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 15:52:00.076918 (+ 248us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:35337: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":334,"threads_started":1}
I20260929 15:52:00.078966 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 5868
I20260929 15:52:00.080017 5929 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:52:00.179242 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 5868
2026-09-29T15:52:00Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1109 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (401 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (408 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:52:01 dist-test-slave-r0ck krb5kdc[5978](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:52:01 dist-test-slave-r0ck krb5kdc[5978](info): set up 2 sockets
Sep 29 15:52:01 dist-test-slave-r0ck krb5kdc[5978](info): commencing operation
krb5kdc: starting...
W20260929 15:52:02.090128 388 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:52:02 dist-test-slave-r0ck krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:52:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:52:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:52:02.285185 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37901
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37901
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:52:02.597223 5991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:52:02.597565 5991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:52:02.597733 5991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:52:02.604838 5991 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:52:02.604977 5991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:52:02.605090 5991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:52:02.605187 5991 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:52:02.605273 5991 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:52:02.615034 5991 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:52:02.616345 5991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:52:02.617668 5991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:52:02.629041 5997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:02.629000 5999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:02.629594 5996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:52:02.630062 5991 server_base.cc:1034] running on GCE node
I20260929 15:52:02.630756 5991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:52:02.632161 5991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:52:02.633505 5991 hybrid_clock.cc:648] HybridClock initialized: now 1790697122633450 us; error 43 us; skew 500 ppm
I20260929 15:52:02.633843 5991 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:52:02 dist-test-slave-r0ck krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:52:02.636729 5991 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:52:02.638152 5991 webserver.cc:460] Webserver started at http://127.0.97.62:46721/ using document root <none> and password file <none>
I20260929 15:52:02.638798 5991 fs_manager.cc:359] Metadata directory not provided
I20260929 15:52:02.638938 5991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:52:02.639233 5991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:52:02.642143 5991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "68be0c3940c347ec98bb636105b982e6"
format_stamp: "Formatted at 2026-09-29 15:52:02 on dist-test-slave-r0ck"
server_key: "d1c7590ee17fe4a871fec80e0e6a324b"
server_key_iv: "9d005da98a25e9faa34fd693399791eb"
server_key_version: "encryptionkey@0"
I20260929 15:52:02.642896 5991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "68be0c3940c347ec98bb636105b982e6"
format_stamp: "Formatted at 2026-09-29 15:52:02 on dist-test-slave-r0ck"
server_key: "d1c7590ee17fe4a871fec80e0e6a324b"
server_key_iv: "9d005da98a25e9faa34fd693399791eb"
server_key_version: "encryptionkey@0"
I20260929 15:52:02.647753 5991 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 15:52:02.651000 6006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:02.652310 5991 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 15:52:02.652483 5991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "68be0c3940c347ec98bb636105b982e6"
format_stamp: "Formatted at 2026-09-29 15:52:02 on dist-test-slave-r0ck"
server_key: "d1c7590ee17fe4a871fec80e0e6a324b"
server_key_iv: "9d005da98a25e9faa34fd693399791eb"
server_key_version: "encryptionkey@0"
I20260929 15:52:02.652667 5991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:52:02.675320 5991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:52:02.679136 5991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:52:02.679421 5991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:52:02.697126 5991 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37901
I20260929 15:52:02.697163 6057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37901 every 8 connection(s)
I20260929 15:52:02.698408 5991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:52:02.701812 6058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:52:02.702889 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 5991
I20260929 15:52:02.703081 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:52:02.703706 388 external_mini_cluster.cc:1428] Setting key fbed7324cb55ce825bd4e22424401861
I20260929 15:52:02.714707 6058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68be0c3940c347ec98bb636105b982e6: Bootstrap starting.
Sep 29 15:52:02 dist-test-slave-r0ck krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:52:02.718385 6058 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68be0c3940c347ec98bb636105b982e6: Neither blocks nor log segments found. Creating new log.
I20260929 15:52:02.719440 6058 log.cc:824] T 00000000000000000000000000000000 P 68be0c3940c347ec98bb636105b982e6: Log is configured to *not* fsync() on all Append() calls
I20260929 15:52:02.722404 6058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68be0c3940c347ec98bb636105b982e6: No bootstrap required, opened a new log
I20260929 15:52:02.726176 6061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:02.705651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38742 (local address 127.0.97.62:37901)
0929 15:52:02.706444 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:02.706458 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:02.706496 (+ 38us) server_negotiation.cc:407] Connection header received
0929 15:52:02.708029 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:02.708053 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:02.708425 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:02.708715 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:02.710426 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:02.711722 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:02.713467 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:02.714098 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:02.714218 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:02.718829 (+ 4611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:02.718840 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:02.718858 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:02.718897 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:02.722091 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:02.722868 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:02.722874 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:02.722880 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:02.723035 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:02.723406 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:02.723415 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:02.723417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:02.723906 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:02.724217 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:02.724639 (+ 422us) server_negotiation.cc:299] Negotiation successful
0929 15:52:02.725025 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":368,"threads_started":1}
I20260929 15:52:02.730288 6058 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68be0c3940c347ec98bb636105b982e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68be0c3940c347ec98bb636105b982e6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37901 } }
I20260929 15:52:02.730633 6058 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68be0c3940c347ec98bb636105b982e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:52:02.730737 6058 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68be0c3940c347ec98bb636105b982e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68be0c3940c347ec98bb636105b982e6, State: Initialized, Role: FOLLOWER
I20260929 15:52:02.731236 6058 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68be0c3940c347ec98bb636105b982e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68be0c3940c347ec98bb636105b982e6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37901 } }
I20260929 15:52:02.731463 6058 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68be0c3940c347ec98bb636105b982e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:52:02.731598 6058 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68be0c3940c347ec98bb636105b982e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:52:02.731758 6058 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68be0c3940c347ec98bb636105b982e6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:52:02.733886 6058 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68be0c3940c347ec98bb636105b982e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68be0c3940c347ec98bb636105b982e6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37901 } }
I20260929 15:52:02.734404 6058 leader_election.cc:302] T 00000000000000000000000000000000 P 68be0c3940c347ec98bb636105b982e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68be0c3940c347ec98bb636105b982e6; no voters:
I20260929 15:52:02.735037 6058 leader_election.cc:288] T 00000000000000000000000000000000 P 68be0c3940c347ec98bb636105b982e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:52:02.735172 6063 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68be0c3940c347ec98bb636105b982e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:52:02.735816 6063 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68be0c3940c347ec98bb636105b982e6 [term 1 LEADER]: Becoming Leader. State: Replica: 68be0c3940c347ec98bb636105b982e6, State: Running, Role: LEADER
I20260929 15:52:02.736378 6063 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68be0c3940c347ec98bb636105b982e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68be0c3940c347ec98bb636105b982e6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37901 } }
I20260929 15:52:02.736909 6058 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68be0c3940c347ec98bb636105b982e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:52:02.739241 6064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68be0c3940c347ec98bb636105b982e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68be0c3940c347ec98bb636105b982e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68be0c3940c347ec98bb636105b982e6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37901 } } }
I20260929 15:52:02.739642 6064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68be0c3940c347ec98bb636105b982e6 [sys.catalog]: This master's current role is: LEADER
I20260929 15:52:02.739303 6065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68be0c3940c347ec98bb636105b982e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68be0c3940c347ec98bb636105b982e6. Latest consensus state: current_term: 1 leader_uuid: "68be0c3940c347ec98bb636105b982e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68be0c3940c347ec98bb636105b982e6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37901 } } }
I20260929 15:52:02.740201 6065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68be0c3940c347ec98bb636105b982e6 [sys.catalog]: This master's current role is: LEADER
I20260929 15:52:02.742429 6072 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:52:02.747452 6072 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:52:02.756496 6072 catalog_manager.cc:1347] Generated new cluster ID: 1604bcbda7824bfdac83e6cfe100f9d5
I20260929 15:52:02.756603 6072 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:52:02.781193 6072 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:52:02.782572 6072 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:52:02.799414 6072 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68be0c3940c347ec98bb636105b982e6: Generated new TSK 0
I20260929 15:52:02.800239 6072 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 15:52:02.858832 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37901
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:52:03.204398 6084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:52:03.204650 6084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:52:03.204824 6084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:52:03.211843 6084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:52:03.211934 6084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:52:03.212154 6084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:52:03.221331 6084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:52:03.222550 6084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:52:03.223830 6084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:52:03.236665 6090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:03.236718 6089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:03.236729 6092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:52:03.238029 6084 server_base.cc:1034] running on GCE node
I20260929 15:52:03.238679 6084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:52:03.239573 6084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:52:03.240782 6084 hybrid_clock.cc:648] HybridClock initialized: now 1790697123240730 us; error 53 us; skew 500 ppm
I20260929 15:52:03.241081 6084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:52:03 dist-test-slave-r0ck krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:52:03.244292 6084 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:52:03.245646 6084 webserver.cc:460] Webserver started at http://127.0.97.1:36325/ using document root <none> and password file <none>
I20260929 15:52:03.246238 6084 fs_manager.cc:359] Metadata directory not provided
I20260929 15:52:03.246341 6084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:52:03.246575 6084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:52:03.249305 6084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-0/data/instance:
uuid: "0e7746c483e042779f6d9d972f513905"
format_stamp: "Formatted at 2026-09-29 15:52:03 on dist-test-slave-r0ck"
server_key: "c45da35bacb9733599b14537405f563e"
server_key_iv: "46cb461754bd119d34e1eda694c7e5d4"
server_key_version: "encryptionkey@0"
I20260929 15:52:03.250137 6084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "0e7746c483e042779f6d9d972f513905"
format_stamp: "Formatted at 2026-09-29 15:52:03 on dist-test-slave-r0ck"
server_key: "c45da35bacb9733599b14537405f563e"
server_key_iv: "46cb461754bd119d34e1eda694c7e5d4"
server_key_version: "encryptionkey@0"
I20260929 15:52:03.254747 6084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 15:52:03.257875 6099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:03.259140 6084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:52:03.259294 6084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-0/wal
uuid: "0e7746c483e042779f6d9d972f513905"
format_stamp: "Formatted at 2026-09-29 15:52:03 on dist-test-slave-r0ck"
server_key: "c45da35bacb9733599b14537405f563e"
server_key_iv: "46cb461754bd119d34e1eda694c7e5d4"
server_key_version: "encryptionkey@0"
I20260929 15:52:03.259496 6084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:52:03.281200 6084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:52:03.284780 6084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:52:03.285090 6084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:52:03.286079 6084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:52:03.287325 6084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:52:03.287428 6084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:03.287576 6084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:52:03.287645 6084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:03.319823 6084 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43903
I20260929 15:52:03.319876 6211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43903 every 8 connection(s)
I20260929 15:52:03.321053 6084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:52:03.331447 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 6084
I20260929 15:52:03.331641 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:52:03.332273 388 external_mini_cluster.cc:1428] Setting key ee7789718693591fb39b6f1d6a757c14
Sep 29 15:52:03 dist-test-slave-r0ck krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:52:03.341506 6214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:03.323585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42267 (local address 127.0.97.62:37901)
0929 15:52:03.324053 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:03.324059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:03.324967 (+ 908us) server_negotiation.cc:407] Connection header received
0929 15:52:03.326746 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:03.326750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:03.326820 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:03.326910 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:03.328290 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:03.329051 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:03.330565 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:03.330946 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:03.331100 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:03.334663 (+ 3563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:03.334672 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:03.334680 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:03.334726 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:03.338254 (+ 3528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:03.338975 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:03.338983 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:03.338989 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:03.339134 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:03.339588 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:03.339595 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:03.339601 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:03.339968 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:03.340113 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:03.340784 (+ 671us) server_negotiation.cc:299] Negotiation successful
0929 15:52:03.341031 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":215,"threads_started":1}
I20260929 15:52:03.342232 6215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:03.324085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37901 (local address 127.0.97.1:42267)
0929 15:52:03.324809 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:03.324853 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:03.326514 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:03.327102 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:03.327109 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:03.327596 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:03.328112 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:03.328127 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:03.329185 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:03.329188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:03.330442 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:03.330453 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:03.331156 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:03.331160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:03.331346 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:03.332108 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:03.332129 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:03.334462 (+ 2333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:03.338431 (+ 3969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:03.338438 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:03.338450 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:03.338841 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:03.339264 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:03.339268 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:03.339271 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:03.339458 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:03.340089 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:03.340094 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:03.340395 (+ 301us) client_negotiation.cc:769] Sending connection context
0929 15:52:03.340613 (+ 218us) client_negotiation.cc:240] Negotiation successful
0929 15:52:03.340977 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":267,"threads_started":1}
I20260929 15:52:03.343807 6212 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37901
I20260929 15:52:03.344102 6212 heartbeater.cc:464] Registering TS with master...
I20260929 15:52:03.344784 6212 heartbeater.cc:511] Master 127.0.97.62:37901 requested a full tablet report, sending...
I20260929 15:52:03.346808 6023 ts_manager.cc:194] Registered new tserver with Master: 0e7746c483e042779f6d9d972f513905 (127.0.97.1:43903)
I20260929 15:52:03.348196 6023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42267
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 15:52:03.379055 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37901
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:52:03.692698 6218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:52:03.692953 6218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:52:03.693100 6218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:52:03.702395 6218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:52:03.702515 6218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:52:03.702690 6218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:52:03.714403 6218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:52:03.715566 6218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:52:03.716791 6218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:52:03.729592 6223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:03.729619 6224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:03.729789 6226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:52:03.730177 6218 server_base.cc:1034] running on GCE node
I20260929 15:52:03.730772 6218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:52:03.731570 6218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:52:03.732839 6218 hybrid_clock.cc:648] HybridClock initialized: now 1790697123732786 us; error 41 us; skew 500 ppm
I20260929 15:52:03.733179 6218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:52:03 dist-test-slave-r0ck krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:52:03.736282 6218 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:52:03.737818 6218 webserver.cc:460] Webserver started at http://127.0.97.2:36105/ using document root <none> and password file <none>
I20260929 15:52:03.738554 6218 fs_manager.cc:359] Metadata directory not provided
I20260929 15:52:03.738692 6218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:52:03.738983 6218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:52:03.741807 6218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-1/data/instance:
uuid: "752a7d5113d249f584403fc627a8b3d5"
format_stamp: "Formatted at 2026-09-29 15:52:03 on dist-test-slave-r0ck"
server_key: "c4ca82d9da2fc5f905385593fffe2cdf"
server_key_iv: "59d84eb1020c738d0853506aa5cf6804"
server_key_version: "encryptionkey@0"
I20260929 15:52:03.742686 6218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "752a7d5113d249f584403fc627a8b3d5"
format_stamp: "Formatted at 2026-09-29 15:52:03 on dist-test-slave-r0ck"
server_key: "c4ca82d9da2fc5f905385593fffe2cdf"
server_key_iv: "59d84eb1020c738d0853506aa5cf6804"
server_key_version: "encryptionkey@0"
I20260929 15:52:03.747691 6218 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 15:52:03.751008 6233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:03.752159 6218 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 15:52:03.752327 6218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-1/wal
uuid: "752a7d5113d249f584403fc627a8b3d5"
format_stamp: "Formatted at 2026-09-29 15:52:03 on dist-test-slave-r0ck"
server_key: "c4ca82d9da2fc5f905385593fffe2cdf"
server_key_iv: "59d84eb1020c738d0853506aa5cf6804"
server_key_version: "encryptionkey@0"
I20260929 15:52:03.752527 6218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:52:03.774446 6218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:52:03.778661 6218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:52:03.778945 6218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:52:03.779838 6218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:52:03.781152 6218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:52:03.781256 6218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:03.781427 6218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:52:03.781505 6218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:03.819133 6218 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40189
I20260929 15:52:03.819188 6345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40189 every 8 connection(s)
I20260929 15:52:03.820360 6218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:52:03.827833 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 6218
I20260929 15:52:03.828015 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:52:03.828423 388 external_mini_cluster.cc:1428] Setting key eee0a8f3f005efd32f127fb9d5d406f5
Sep 29 15:52:03 dist-test-slave-r0ck krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:52:03.839704 6214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:03.822827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51775 (local address 127.0.97.62:37901)
0929 15:52:03.822993 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:03.822997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:03.824066 (+ 1069us) server_negotiation.cc:407] Connection header received
0929 15:52:03.825813 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:03.825819 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:03.825914 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:03.826023 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:03.827402 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:03.828212 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:03.829908 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:03.830342 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:03.830474 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:03.834008 (+ 3534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:03.834012 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:03.834016 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:03.834044 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:03.836490 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:03.837422 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:03.837426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:03.837429 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:03.837574 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:03.838012 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:03.838016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:03.838018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:03.838266 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:03.838413 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:03.839230 (+ 817us) server_negotiation.cc:299] Negotiation successful
0929 15:52:03.839390 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 15:52:03.841102 6348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:03.823248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37901 (local address 127.0.97.2:51775)
0929 15:52:03.823918 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:03.823952 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:03.825540 (+ 1588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:03.826235 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:03.826243 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:03.826722 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:03.827242 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:03.827257 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:03.828358 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:03.828362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:03.829766 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:03.829778 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:03.830559 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:03.830565 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:03.830783 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:03.831475 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:03.831497 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:03.833794 (+ 2297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:03.836625 (+ 2831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:03.836633 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:03.836648 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:03.837258 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:03.837665 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:03.837673 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:03.837679 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:03.837898 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:03.838414 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:03.838424 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:03.838853 (+ 429us) client_negotiation.cc:769] Sending connection context
0929 15:52:03.839140 (+ 287us) client_negotiation.cc:240] Negotiation successful
0929 15:52:03.839714 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":269,"threads_started":1}
I20260929 15:52:03.842551 6346 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37901
I20260929 15:52:03.842897 6346 heartbeater.cc:464] Registering TS with master...
I20260929 15:52:03.843652 6346 heartbeater.cc:511] Master 127.0.97.62:37901 requested a full tablet report, sending...
I20260929 15:52:03.845140 6023 ts_manager.cc:194] Registered new tserver with Master: 752a7d5113d249f584403fc627a8b3d5 (127.0.97.2:40189)
I20260929 15:52:03.845981 6023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:51775
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 15:52:03.874982 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37901
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:52:04.185669 6351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:52:04.185961 6351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:52:04.186129 6351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:52:04.193202 6351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:52:04.193363 6351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:52:04.193534 6351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:52:04.202956 6351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:52:04.204141 6351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:52:04.205366 6351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:52:04.217486 6356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:04.217572 6357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:04.217527 6359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:52:04.218767 6351 server_base.cc:1034] running on GCE node
I20260929 15:52:04.219288 6351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:52:04.220170 6351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:52:04.221428 6351 hybrid_clock.cc:648] HybridClock initialized: now 1790697124221324 us; error 35 us; skew 500 ppm
I20260929 15:52:04.221711 6351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:52:04 dist-test-slave-r0ck krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:52:04.224560 6351 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:52:04.225948 6351 webserver.cc:460] Webserver started at http://127.0.97.3:38891/ using document root <none> and password file <none>
I20260929 15:52:04.226559 6351 fs_manager.cc:359] Metadata directory not provided
I20260929 15:52:04.226679 6351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:52:04.226974 6351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:52:04.229722 6351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-2/data/instance:
uuid: "b3d3562623164691845fe29e2fadc762"
format_stamp: "Formatted at 2026-09-29 15:52:04 on dist-test-slave-r0ck"
server_key: "1b2f2880896cd2cd09bf8c3b54247815"
server_key_iv: "2e224da8f4be724c4f9473558ef1eb9e"
server_key_version: "encryptionkey@0"
I20260929 15:52:04.230445 6351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b3d3562623164691845fe29e2fadc762"
format_stamp: "Formatted at 2026-09-29 15:52:04 on dist-test-slave-r0ck"
server_key: "1b2f2880896cd2cd09bf8c3b54247815"
server_key_iv: "2e224da8f4be724c4f9473558ef1eb9e"
server_key_version: "encryptionkey@0"
I20260929 15:52:04.235019 6351 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 15:52:04.238372 6366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:04.239583 6351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:52:04.239744 6351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-2/wal
uuid: "b3d3562623164691845fe29e2fadc762"
format_stamp: "Formatted at 2026-09-29 15:52:04 on dist-test-slave-r0ck"
server_key: "1b2f2880896cd2cd09bf8c3b54247815"
server_key_iv: "2e224da8f4be724c4f9473558ef1eb9e"
server_key_version: "encryptionkey@0"
I20260929 15:52:04.239928 6351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:52:04.254019 6351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:52:04.257938 6351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:52:04.258194 6351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:52:04.259025 6351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:52:04.260208 6351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:52:04.260279 6351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:04.260406 6351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:52:04.260457 6351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:04.293004 6351 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38203
I20260929 15:52:04.293076 6478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38203 every 8 connection(s)
I20260929 15:52:04.294255 6351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:52:04.302865 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 6351
I20260929 15:52:04.303030 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:52:04.303550 388 external_mini_cluster.cc:1428] Setting key 310502aaa346f8e72395a6117e0e523f
Sep 29 15:52:04 dist-test-slave-r0ck krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:52:04.313268 6214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:04.296746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52175 (local address 127.0.97.62:37901)
0929 15:52:04.296915 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:04.296919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:04.298006 (+ 1087us) server_negotiation.cc:407] Connection header received
0929 15:52:04.299685 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:04.299690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:04.299778 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:04.299885 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:04.301260 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:04.302057 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:04.304212 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:04.304715 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:04.304838 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:04.308086 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:04.308097 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:04.308101 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:04.308129 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:04.310517 (+ 2388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:04.311169 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:04.311176 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:04.311179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:04.311303 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:04.311711 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:04.311716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:04.311719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:04.311959 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:04.312105 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:04.312725 (+ 620us) server_negotiation.cc:299] Negotiation successful
0929 15:52:04.312926 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 15:52:04.314136 6481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:04.297099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37901 (local address 127.0.97.3:52175)
0929 15:52:04.297830 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:04.297870 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:04.299436 (+ 1566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:04.300125 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:04.300133 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:04.300587 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:04.301094 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:04.301110 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:04.302208 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:04.302212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:04.304072 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:04.304083 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:04.304925 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:04.304933 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:04.305169 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:04.305851 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:04.305872 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:04.307877 (+ 2005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:04.310649 (+ 2772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:04.310657 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:04.310671 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:04.311054 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:04.311418 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:04.311422 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:04.311426 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:04.311595 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:04.312078 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:04.312085 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:04.312378 (+ 293us) client_negotiation.cc:769] Sending connection context
0929 15:52:04.312633 (+ 255us) client_negotiation.cc:240] Negotiation successful
0929 15:52:04.313117 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":344,"threads_started":1}
I20260929 15:52:04.315446 6479 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37901
I20260929 15:52:04.315714 6479 heartbeater.cc:464] Registering TS with master...
I20260929 15:52:04.316342 6479 heartbeater.cc:511] Master 127.0.97.62:37901 requested a full tablet report, sending...
I20260929 15:52:04.317660 6023 ts_manager.cc:194] Registered new tserver with Master: b3d3562623164691845fe29e2fadc762 (127.0.97.3:38203)
I20260929 15:52:04.318493 6023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52175
I20260929 15:52:04.318657 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/krb5kdc/test-admin.keytab.
Sep 29 15:52:04 dist-test-slave-r0ck krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:52:04.346730 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
I20260929 15:52:04.350931 6212 heartbeater.cc:503] Master 127.0.97.62:37901 was elected leader, sending a full tablet report...
W20260929 15:52:04.351706 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/krb5kdc/krb5cc' of type FILE
W20260929 15:52:04.352582 6486 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:04.349260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57866 (local address 127.0.97.1:43903)
0929 15:52:04.349929 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:04.349936 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:04.349959 (+ 23us) server_negotiation.cc:407] Connection header received
0929 15:52:04.350016 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:04.350023 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:04.350215 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:04.350377 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:04.351308 (+ 931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57866: BlockingRecv error: recv got EOF from 127.0.0.1:57866 (error 108)
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":325,"threads_started":1}
Sep 29 15:52:04 dist-test-slave-r0ck krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:52:04.353432 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 15:52:04 dist-test-slave-r0ck krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:52:04.354766 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 15:52:04 dist-test-slave-r0ck krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 15:52:04.369393 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:04.356324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57868 (local address 127.0.97.1:43903)
0929 15:52:04.356523 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:04.356529 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:04.356856 (+ 327us) server_negotiation.cc:407] Connection header received
0929 15:52:04.357018 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:04.357022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:04.357089 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:04.357160 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:04.358145 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:04.359053 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:04.360173 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:04.360699 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:04.360788 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:04.364163 (+ 3375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:04.364172 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:04.364185 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:04.364216 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:04.366757 (+ 2541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:04.367359 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:04.367366 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:04.367368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:04.367514 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:04.367926 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:04.367930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:04.367932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:04.368277 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:04.368497 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:04.368779 (+ 282us) server_negotiation.cc:299] Negotiation successful
0929 15:52:04.369083 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 15:52:04.374720 6486 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:04.373080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57884 (local address 127.0.97.1:43903)
0929 15:52:04.373293 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:04.373298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:04.373531 (+ 233us) server_negotiation.cc:407] Connection header received
0929 15:52:04.373700 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:04.373711 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:04.373761 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:04.373857 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:04.374504 (+ 647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57884: BlockingRecv error: recv got EOF from 127.0.0.1:57884 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260929 15:52:04.374949 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/krb5kdc/krb5cc' of type FILE
Sep 29 15:52:04 dist-test-slave-r0ck krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:52:04.376302 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 15:52:04 dist-test-slave-r0ck krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:52:04.377584 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 15:52:04 dist-test-slave-r0ck krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 15:52:04.390578 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:04.378826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57890 (local address 127.0.97.1:43903)
0929 15:52:04.379046 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:04.379051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:04.379229 (+ 178us) server_negotiation.cc:407] Connection header received
0929 15:52:04.379329 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:04.379332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:04.379378 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:04.379450 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:04.380249 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:04.380888 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:04.382043 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:04.382443 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:04.382531 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:04.385841 (+ 3310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:04.385845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:04.385848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:04.385887 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:04.388421 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:04.389028 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:04.389032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:04.389034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:04.389148 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:04.389508 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:04.389511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:04.389513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:04.389751 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:04.389937 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:04.390123 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 15:52:04.390296 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260929 15:52:04.394250 6486 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:04.392754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57898 (local address 127.0.97.1:43903)
0929 15:52:04.392936 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:04.392941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:04.393113 (+ 172us) server_negotiation.cc:407] Connection header received
0929 15:52:04.393204 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:04.393207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:04.393265 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:04.393336 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:04.394037 (+ 701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57898: BlockingRecv error: recv got EOF from 127.0.0.1:57898 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 15:52:04.394502 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697075657932-388-0/krb5kdc/krb5cc' of type FILE
Sep 29 15:52:04 dist-test-slave-r0ck krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:52:04.395877 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 15:52:04 dist-test-slave-r0ck krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:52:04.397065 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 15:52:04 dist-test-slave-r0ck krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 15:52:04.410439 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:04.398356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57912 (local address 127.0.97.1:43903)
0929 15:52:04.398556 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:04.398563 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:04.398747 (+ 184us) server_negotiation.cc:407] Connection header received
0929 15:52:04.398948 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:04.398953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:04.399012 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:04.399110 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:04.400053 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:04.400774 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:04.402009 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:04.402419 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:04.402542 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:04.405753 (+ 3211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:04.405758 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:04.405761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:04.405788 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:04.408220 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:04.408756 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:04.408759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:04.408762 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:04.408896 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:04.409220 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:04.409223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:04.409225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:04.409528 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:04.409743 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:04.409977 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 15:52:04.410149 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 15:52:04.411991 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 6084
I20260929 15:52:04.422696 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:04.412626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57920 (local address 127.0.97.1:43903)
0929 15:52:04.412753 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:04.412757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:04.413110 (+ 353us) server_negotiation.cc:407] Connection header received
0929 15:52:04.413225 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:04.413228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:04.413273 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:04.413393 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:04.414192 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:04.414823 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:04.415988 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:04.416364 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:04.416449 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:04.418322 (+ 1873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:04.418326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:04.418329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:04.418354 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:04.420662 (+ 2308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:04.421224 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:04.421227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:04.421229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:04.421334 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:04.421686 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:04.421689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:04.421690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:04.421897 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:04.422075 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:04.422305 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 15:52:04.422449 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 15:52:04.423215 6207 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:52:04.506903 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 6084
I20260929 15:52:04.522871 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 6218
Sep 29 15:52:04 dist-test-slave-r0ck krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 15:52:04.537784 6499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:04.523737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56908 (local address 127.0.97.2:40189)
0929 15:52:04.524417 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:04.524426 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:04.524453 (+ 27us) server_negotiation.cc:407] Connection header received
0929 15:52:04.524555 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:04.524567 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:04.524861 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:04.525028 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:04.526027 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:04.527018 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:04.528248 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:04.528790 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:04.528925 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:04.532667 (+ 3742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:04.532677 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:04.532690 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:04.532722 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:04.535385 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:04.536016 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:04.536023 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:04.536026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:04.536136 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:04.536458 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:04.536460 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:04.536462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:04.536770 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:04.536998 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:04.537269 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 15:52:04.537507 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":346,"threads_started":1}
I20260929 15:52:04.539039 6341 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:52:04.620963 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 6218
I20260929 15:52:04.633543 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 6351
Sep 29 15:52:04 dist-test-slave-r0ck krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 15:52:04.648846 6501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:04.634506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58604 (local address 127.0.97.3:38203)
0929 15:52:04.635109 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:04.635116 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:04.635143 (+ 27us) server_negotiation.cc:407] Connection header received
0929 15:52:04.635200 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:04.635206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:04.635406 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:04.635589 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:04.636638 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:04.637751 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:04.639077 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:04.639616 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:04.639753 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:04.643796 (+ 4043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:04.643805 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:04.643820 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:04.643851 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:04.646583 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:04.647178 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:04.647184 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:04.647186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:04.647294 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:04.647595 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:04.647598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:04.647600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:04.647898 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:04.648126 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:04.648365 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 15:52:04.648547 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":297,"threads_started":1}
I20260929 15:52:04.650195 6474 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:52:04.731259 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 6351
I20260929 15:52:04.747102 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 5991
I20260929 15:52:04.748243 6053 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:52:04.848533 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 5991
2026-09-29T15:52:04Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3863 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:52:04 dist-test-slave-r0ck krb5kdc[6506](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:52:04 dist-test-slave-r0ck krb5kdc[6506](info): set up 2 sockets
Sep 29 15:52:04 dist-test-slave-r0ck krb5kdc[6506](info): commencing operation
krb5kdc: starting...
W20260929 15:52:05.942157 388 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:52:06 dist-test-slave-r0ck krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:52:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:52:06Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260929 15:52:06.156386 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33067
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33067
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:52:06.466403 6519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:52:06.466753 6519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:52:06.466907 6519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:52:06.474476 6519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:52:06.474628 6519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:52:06.474741 6519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:52:06.474836 6519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:52:06.474941 6519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:52:06.485113 6519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:52:06.486660 6519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:52:06.488022 6519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:52:06.499382 6527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:06.499480 6525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:06.499472 6524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:52:06.500612 6519 server_base.cc:1034] running on GCE node
I20260929 15:52:06.501458 6519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:52:06.502843 6519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:52:06.504087 6519 hybrid_clock.cc:648] HybridClock initialized: now 1790697126504033 us; error 50 us; skew 500 ppm
I20260929 15:52:06.504454 6519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:52:06 dist-test-slave-r0ck krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697126, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:52:06.510799 6519 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260929 15:52:06.512266 6519 webserver.cc:460] Webserver started at http://127.0.97.62:42283/ using document root <none> and password file <none>
I20260929 15:52:06.512966 6519 fs_manager.cc:359] Metadata directory not provided
I20260929 15:52:06.513110 6519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:52:06.513504 6519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:52:06.516373 6519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "e21605ed4e8e4e9189d58164059f7de4"
format_stamp: "Formatted at 2026-09-29 15:52:06 on dist-test-slave-r0ck"
server_key: "fac12a54b57644e91ecb0a5e82bce714"
server_key_iv: "2011f01c69dee75514da78ec033548f4"
server_key_version: "encryptionkey@0"
I20260929 15:52:06.517117 6519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "e21605ed4e8e4e9189d58164059f7de4"
format_stamp: "Formatted at 2026-09-29 15:52:06 on dist-test-slave-r0ck"
server_key: "fac12a54b57644e91ecb0a5e82bce714"
server_key_iv: "2011f01c69dee75514da78ec033548f4"
server_key_version: "encryptionkey@0"
I20260929 15:52:06.522132 6519 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.002s
I20260929 15:52:06.525861 6534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:06.527310 6519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 15:52:06.527508 6519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "e21605ed4e8e4e9189d58164059f7de4"
format_stamp: "Formatted at 2026-09-29 15:52:06 on dist-test-slave-r0ck"
server_key: "fac12a54b57644e91ecb0a5e82bce714"
server_key_iv: "2011f01c69dee75514da78ec033548f4"
server_key_version: "encryptionkey@0"
I20260929 15:52:06.527699 6519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:52:06.557649 6519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:52:06.561975 6519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:52:06.562294 6519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:52:06.581396 6519 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33067
I20260929 15:52:06.581454 6585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33067 every 8 connection(s)
I20260929 15:52:06.582758 6519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:52:06.585101 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 6519
I20260929 15:52:06.585309 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:52:06.585954 388 external_mini_cluster.cc:1428] Setting key d0eb007e9f5c6ec334e12074a896cd3e
I20260929 15:52:06.586547 6586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:52:06.598165 6586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4: Bootstrap starting.
Sep 29 15:52:06 dist-test-slave-r0ck krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 15:52:06.601537 6586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4: Neither blocks nor log segments found. Creating new log.
I20260929 15:52:06.602711 6586 log.cc:824] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4: Log is configured to *not* fsync() on all Append() calls
I20260929 15:52:06.605590 6586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4: No bootstrap required, opened a new log
I20260929 15:52:06.608212 6589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:06.587896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42512 (local address 127.0.97.62:33067)
0929 15:52:06.588636 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:06.588648 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:06.588735 (+ 87us) server_negotiation.cc:407] Connection header received
0929 15:52:06.591007 (+ 2272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:06.591027 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:06.591389 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:06.591758 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:06.593148 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:06.594439 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:06.595705 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:06.596398 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:06.596592 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:06.600862 (+ 4270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:06.600870 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:06.600884 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:06.600922 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:06.604341 (+ 3419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:06.604993 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:06.605008 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:06.605014 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:06.605167 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:06.605603 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:06.605608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:06.605613 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:06.606151 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:06.606456 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:06.606857 (+ 401us) server_negotiation.cc:299] Negotiation successful
0929 15:52:06.607277 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":330,"threads_started":1}
I20260929 15:52:06.613631 6586 raft_consensus.cc:348] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } }
I20260929 15:52:06.614018 6586 raft_consensus.cc:374] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:52:06.614135 6586 raft_consensus.cc:729] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e21605ed4e8e4e9189d58164059f7de4, State: Initialized, Role: FOLLOWER
I20260929 15:52:06.614715 6586 consensus_queue.cc:260] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } }
I20260929 15:52:06.614950 6586 raft_consensus.cc:388] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:52:06.615089 6586 raft_consensus.cc:482] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:52:06.615267 6586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:52:06.617491 6586 raft_consensus.cc:504] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } }
I20260929 15:52:06.617964 6586 leader_election.cc:302] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e21605ed4e8e4e9189d58164059f7de4; no voters:
I20260929 15:52:06.618510 6586 leader_election.cc:288] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:52:06.618655 6591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:52:06.619315 6591 raft_consensus.cc:686] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [term 1 LEADER]: Becoming Leader. State: Replica: e21605ed4e8e4e9189d58164059f7de4, State: Running, Role: LEADER
I20260929 15:52:06.619791 6591 consensus_queue.cc:237] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } }
I20260929 15:52:06.620469 6586 sys_catalog.cc:564] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:52:06.622511 6593 sys_catalog.cc:455] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e21605ed4e8e4e9189d58164059f7de4. Latest consensus state: current_term: 1 leader_uuid: "e21605ed4e8e4e9189d58164059f7de4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } }
I20260929 15:52:06.622946 6593 sys_catalog.cc:458] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [sys.catalog]: This master's current role is: LEADER
I20260929 15:52:06.623178 6592 sys_catalog.cc:455] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e21605ed4e8e4e9189d58164059f7de4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } }
I20260929 15:52:06.623555 6592 sys_catalog.cc:458] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [sys.catalog]: This master's current role is: LEADER
I20260929 15:52:06.624955 6598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:52:06.629333 6598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:52:06.637714 6598 catalog_manager.cc:1347] Generated new cluster ID: 157dc853f93b4db599b14cf25dcddf85
I20260929 15:52:06.637821 6598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:52:06.656309 6598 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:52:06.658136 6598 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:52:06.666483 6598 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4: Generated new TSK 0
I20260929 15:52:06.667481 6598 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260929 15:52:06.716617 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33067
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:52:07.037261 6612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:52:07.037530 6612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:52:07.037716 6612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:52:07.044909 6612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:52:07.045017 6612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:52:07.045215 6612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:52:07.054749 6612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:52:07.055979 6612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:52:07.057235 6612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:52:07.070115 6618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:07.070165 6620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:07.070189 6617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:52:07.070960 6612 server_base.cc:1034] running on GCE node
I20260929 15:52:07.071518 6612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:52:07.072376 6612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:52:07.073596 6612 hybrid_clock.cc:648] HybridClock initialized: now 1790697127073541 us; error 45 us; skew 500 ppm
I20260929 15:52:07.073904 6612 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:52:07 dist-test-slave-r0ck krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697127, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:52:07.076848 6612 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260929 15:52:07.078197 6612 webserver.cc:460] Webserver started at http://127.0.97.1:43383/ using document root <none> and password file <none>
I20260929 15:52:07.078831 6612 fs_manager.cc:359] Metadata directory not provided
I20260929 15:52:07.078970 6612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:52:07.079252 6612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:52:07.082059 6612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-0/data/instance:
uuid: "3ab5024e8d1a4d259eefb3f421e119fc"
format_stamp: "Formatted at 2026-09-29 15:52:07 on dist-test-slave-r0ck"
server_key: "bd7b0be039b5009e9e37a85e03f0d97e"
server_key_iv: "def5095a066d96cf95a2a94b2741a749"
server_key_version: "encryptionkey@0"
I20260929 15:52:07.082813 6612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3ab5024e8d1a4d259eefb3f421e119fc"
format_stamp: "Formatted at 2026-09-29 15:52:07 on dist-test-slave-r0ck"
server_key: "bd7b0be039b5009e9e37a85e03f0d97e"
server_key_iv: "def5095a066d96cf95a2a94b2741a749"
server_key_version: "encryptionkey@0"
I20260929 15:52:07.087966 6612 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 15:52:07.091385 6627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:07.092597 6612 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 15:52:07.092765 6612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-0/wal
uuid: "3ab5024e8d1a4d259eefb3f421e119fc"
format_stamp: "Formatted at 2026-09-29 15:52:07 on dist-test-slave-r0ck"
server_key: "bd7b0be039b5009e9e37a85e03f0d97e"
server_key_iv: "def5095a066d96cf95a2a94b2741a749"
server_key_version: "encryptionkey@0"
I20260929 15:52:07.092990 6612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:52:07.110090 6612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:52:07.114555 6612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:52:07.114854 6612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:52:07.115764 6612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:52:07.117035 6612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:52:07.117106 6612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:07.117269 6612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:52:07.117327 6612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:07.147902 6612 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43917
I20260929 15:52:07.147948 6739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43917 every 8 connection(s)
I20260929 15:52:07.149106 6612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:52:07.156095 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 6612
I20260929 15:52:07.156292 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:52:07.156903 388 external_mini_cluster.cc:1428] Setting key 975121ca139f2ab4b41d827429daf354
Sep 29 15:52:07 dist-test-slave-r0ck krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697127, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 15:52:07.169752 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:07.151575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44379 (local address 127.0.97.62:33067)
0929 15:52:07.152036 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:07.152042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:07.152900 (+ 858us) server_negotiation.cc:407] Connection header received
0929 15:52:07.154660 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:07.154665 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:07.154728 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:07.154823 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:07.156443 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:07.157233 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:07.159165 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:07.159534 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:07.159627 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:07.163684 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:07.163689 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:07.163692 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:07.163722 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:07.166583 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:07.167491 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:07.167495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:07.167498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:07.167603 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:07.168071 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:07.168073 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:07.168075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:07.168347 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:07.168450 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:07.169302 (+ 852us) server_negotiation.cc:299] Negotiation successful
0929 15:52:07.169507 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":222,"threads_started":1}
I20260929 15:52:07.171190 6743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:07.152069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33067 (local address 127.0.97.1:44379)
0929 15:52:07.152751 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:07.152787 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:07.154413 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:07.155017 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:07.155024 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:07.155473 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:07.156254 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:07.156281 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:07.157418 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:07.157423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:07.159022 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:07.159037 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:07.159751 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:07.159761 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:07.160116 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:07.160827 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:07.160863 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:07.163461 (+ 2598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:07.166748 (+ 3287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:07.166759 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:07.166778 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:07.167368 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:07.167712 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:07.167719 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:07.167726 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:07.167960 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:07.168476 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:07.168487 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:07.168920 (+ 433us) client_negotiation.cc:769] Sending connection context
0929 15:52:07.169219 (+ 299us) client_negotiation.cc:240] Negotiation successful
0929 15:52:07.169794 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":270,"threads_started":1}
I20260929 15:52:07.172626 6740 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33067
I20260929 15:52:07.172920 6740 heartbeater.cc:464] Registering TS with master...
I20260929 15:52:07.173646 6740 heartbeater.cc:511] Master 127.0.97.62:33067 requested a full tablet report, sending...
I20260929 15:52:07.175486 6551 ts_manager.cc:194] Registered new tserver with Master: 3ab5024e8d1a4d259eefb3f421e119fc (127.0.97.1:43917)
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 15:52:07.176872 6551 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:44379
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260929 15:52:07.202452 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33067
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:52:07.518317 6746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:52:07.518575 6746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:52:07.518671 6746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:52:07.525895 6746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:52:07.526037 6746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:52:07.526163 6746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:52:07.535629 6746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:52:07.536844 6746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:52:07.538126 6746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:52:07.551476 6751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:07.551400 6754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:07.551437 6752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:52:07.552702 6746 server_base.cc:1034] running on GCE node
I20260929 15:52:07.553284 6746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:52:07.554203 6746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:52:07.555430 6746 hybrid_clock.cc:648] HybridClock initialized: now 1790697127555388 us; error 39 us; skew 500 ppm
I20260929 15:52:07.555743 6746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:52:07 dist-test-slave-r0ck krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697127, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:52:07.558844 6746 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260929 15:52:07.560277 6746 webserver.cc:460] Webserver started at http://127.0.97.2:44537/ using document root <none> and password file <none>
I20260929 15:52:07.560945 6746 fs_manager.cc:359] Metadata directory not provided
I20260929 15:52:07.561088 6746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:52:07.561450 6746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:52:07.564225 6746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-1/data/instance:
uuid: "2f0aa7392be545f89274d4a92743c022"
format_stamp: "Formatted at 2026-09-29 15:52:07 on dist-test-slave-r0ck"
server_key: "4f91533c312bbd4195c9ed1f9955e2f5"
server_key_iv: "1665efa9fdc040066736d1de10e14c72"
server_key_version: "encryptionkey@0"
I20260929 15:52:07.564994 6746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2f0aa7392be545f89274d4a92743c022"
format_stamp: "Formatted at 2026-09-29 15:52:07 on dist-test-slave-r0ck"
server_key: "4f91533c312bbd4195c9ed1f9955e2f5"
server_key_iv: "1665efa9fdc040066736d1de10e14c72"
server_key_version: "encryptionkey@0"
I20260929 15:52:07.570075 6746 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260929 15:52:07.573302 6761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:07.574545 6746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 15:52:07.574718 6746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-1/wal
uuid: "2f0aa7392be545f89274d4a92743c022"
format_stamp: "Formatted at 2026-09-29 15:52:07 on dist-test-slave-r0ck"
server_key: "4f91533c312bbd4195c9ed1f9955e2f5"
server_key_iv: "1665efa9fdc040066736d1de10e14c72"
server_key_version: "encryptionkey@0"
I20260929 15:52:07.574906 6746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:52:07.585596 6746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:52:07.589274 6746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:52:07.589588 6746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:52:07.590529 6746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:52:07.591811 6746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:52:07.591912 6746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:07.592052 6746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:52:07.592124 6746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:07.625578 6746 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42699
I20260929 15:52:07.625626 6873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42699 every 8 connection(s)
I20260929 15:52:07.626909 6746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:52:07.630947 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 6746
I20260929 15:52:07.631116 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:52:07.631574 388 external_mini_cluster.cc:1428] Setting key 65bb79161b01976bbfe3c735b37fc8df
Sep 29 15:52:07 dist-test-slave-r0ck krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697127, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 15:52:07.647348 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:07.629532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59467 (local address 127.0.97.62:33067)
0929 15:52:07.629718 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:07.629722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:07.631000 (+ 1278us) server_negotiation.cc:407] Connection header received
0929 15:52:07.633461 (+ 2461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:07.633466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:07.633548 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:07.633663 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:07.635067 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:07.635767 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:07.637319 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:07.637871 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:07.638041 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:07.641757 (+ 3716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:07.641763 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:07.641766 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:07.641795 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:07.644362 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:07.645084 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:07.645090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:07.645094 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:07.645233 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:07.645673 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:07.645680 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:07.645683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:07.645939 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:07.646091 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:07.646774 (+ 683us) server_negotiation.cc:299] Negotiation successful
0929 15:52:07.647002 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 15:52:07.648301 6876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:07.630004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33067 (local address 127.0.97.2:59467)
0929 15:52:07.630833 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:07.630891 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:07.633151 (+ 2260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:07.633888 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:07.633897 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:07.634357 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:07.634910 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:07.634927 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:07.635916 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:07.635922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:07.637170 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:07.637182 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:07.638123 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:07.638131 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:07.638365 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:07.639058 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:07.639080 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:07.641542 (+ 2462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:07.644518 (+ 2976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:07.644526 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:07.644543 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:07.644961 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:07.645374 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:07.645379 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:07.645383 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:07.645561 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:07.646072 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:07.646080 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:07.646415 (+ 335us) client_negotiation.cc:769] Sending connection context
0929 15:52:07.646680 (+ 265us) client_negotiation.cc:240] Negotiation successful
0929 15:52:07.647202 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":289,"threads_started":1}
I20260929 15:52:07.649822 6874 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33067
I20260929 15:52:07.650182 6874 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 15:52:07.651137 6874 heartbeater.cc:511] Master 127.0.97.62:33067 requested a full tablet report, sending...
I20260929 15:52:07.652642 6551 ts_manager.cc:194] Registered new tserver with Master: 2f0aa7392be545f89274d4a92743c022 (127.0.97.2:42699)
I20260929 15:52:07.653551 6551 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59467
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260929 15:52:07.680749 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33067
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:52:08.001837 6879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:52:08.002104 6879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:52:08.002194 6879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:52:08.009289 6879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:52:08.009485 6879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:52:08.009617 6879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:52:08.019311 6879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:52:08.020571 6879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:52:08.021992 6879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:52:08.034643 6884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:08.034752 6885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:08.034756 6887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:52:08.035538 6879 server_base.cc:1034] running on GCE node
I20260929 15:52:08.036499 6879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:52:08.037802 6879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:52:08.039065 6879 hybrid_clock.cc:648] HybridClock initialized: now 1790697128038978 us; error 90 us; skew 500 ppm
I20260929 15:52:08.039510 6879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:52:08 dist-test-slave-r0ck krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697128, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:52:08.042834 6879 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260929 15:52:08.044482 6879 webserver.cc:460] Webserver started at http://127.0.97.3:40793/ using document root <none> and password file <none>
I20260929 15:52:08.045331 6879 fs_manager.cc:359] Metadata directory not provided
I20260929 15:52:08.045517 6879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:52:08.045769 6879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:52:08.048637 6879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-2/data/instance:
uuid: "6b634f743a52484a9f8f7978e146cbba"
format_stamp: "Formatted at 2026-09-29 15:52:08 on dist-test-slave-r0ck"
server_key: "a890323a08047d200c12c74e2f3eb704"
server_key_iv: "c7fdcf9e87ba04f082bf5491f056e6ab"
server_key_version: "encryptionkey@0"
I20260929 15:52:08.049573 6879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6b634f743a52484a9f8f7978e146cbba"
format_stamp: "Formatted at 2026-09-29 15:52:08 on dist-test-slave-r0ck"
server_key: "a890323a08047d200c12c74e2f3eb704"
server_key_iv: "c7fdcf9e87ba04f082bf5491f056e6ab"
server_key_version: "encryptionkey@0"
I20260929 15:52:08.054742 6879 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 15:52:08.058439 6894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:08.059970 6879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:52:08.060149 6879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-2/wal
uuid: "6b634f743a52484a9f8f7978e146cbba"
format_stamp: "Formatted at 2026-09-29 15:52:08 on dist-test-slave-r0ck"
server_key: "a890323a08047d200c12c74e2f3eb704"
server_key_iv: "c7fdcf9e87ba04f082bf5491f056e6ab"
server_key_version: "encryptionkey@0"
I20260929 15:52:08.060357 6879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:52:08.083436 6879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:52:08.087885 6879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:52:08.088186 6879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:52:08.089184 6879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:52:08.090590 6879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:52:08.090693 6879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:08.090845 6879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:52:08.090916 6879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:08.124374 6879 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35181
I20260929 15:52:08.124455 7006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35181 every 8 connection(s)
I20260929 15:52:08.126148 6879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:52:08.134948 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 6879
I20260929 15:52:08.135143 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:52:08.135658 388 external_mini_cluster.cc:1428] Setting key 82ba1810222e570a2638ed6405149d2e
Sep 29 15:52:08 dist-test-slave-r0ck krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697128, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 15:52:08.154804 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:08.136963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33175 (local address 127.0.97.62:33067)
0929 15:52:08.138739 (+ 1776us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:08.138747 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:08.138770 (+ 23us) server_negotiation.cc:407] Connection header received
0929 15:52:08.139810 (+ 1040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:08.139815 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:08.139893 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:08.140020 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:08.141940 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:08.142695 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:08.144201 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:08.144616 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:08.144748 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:08.148336 (+ 3588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:08.148341 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:08.148344 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:08.148373 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:08.151523 (+ 3150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:08.152393 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:08.152398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:08.152403 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:08.152561 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:08.153038 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:08.153043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:08.153047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:08.153321 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:08.153504 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:08.154207 (+ 703us) server_negotiation.cc:299] Negotiation successful
0929 15:52:08.154462 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1626}
I20260929 15:52:08.155761 7009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:08.135670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33067 (local address 127.0.97.3:33175)
0929 15:52:08.136587 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:08.136642 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:08.139486 (+ 2844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:08.140245 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:08.140257 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:08.141048 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:08.141779 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:08.141803 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:08.142842 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:08.142848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:08.144053 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:08.144064 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:08.144842 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:08.144849 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:08.145087 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:08.145819 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:08.145854 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:08.148120 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:08.151687 (+ 3567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:08.151695 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:08.151710 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:08.152267 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:08.152701 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:08.152705 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:08.152709 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:08.152907 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:08.153561 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:08.153567 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:08.153846 (+ 279us) client_negotiation.cc:769] Sending connection context
0929 15:52:08.154104 (+ 258us) client_negotiation.cc:240] Negotiation successful
0929 15:52:08.154598 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":1701,"threads_started":1}
I20260929 15:52:08.157099 7007 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33067
I20260929 15:52:08.157456 7007 heartbeater.cc:464] Registering TS with master...
I20260929 15:52:08.158138 7007 heartbeater.cc:511] Master 127.0.97.62:33067 requested a full tablet report, sending...
I20260929 15:52:08.159583 6551 ts_manager.cc:194] Registered new tserver with Master: 6b634f743a52484a9f8f7978e146cbba (127.0.97.3:35181)
I20260929 15:52:08.160377 6551 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33175
I20260929 15:52:08.165119 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:52:08.179553 6740 heartbeater.cc:503] Master 127.0.97.62:33067 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260929 15:52:08.212363 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:41069
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33067,127.0.97.61:41069
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:52:08.531452 7012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:52:08.531759 7012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:52:08.531898 7012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:52:08.539506 7012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:52:08.539599 7012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:52:08.539644 7012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:52:08.539702 7012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:52:08.539744 7012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:52:08.548944 7012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33067,127.0.97.61:41069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:41069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
W20260929 15:52:08.549206 7012 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:33067,127.0.97.61:41069'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 15:52:08.550163 7012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:52:08.551338 7012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:52:08.562422 7020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:08.562527 7017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:08.562563 7018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:52:08.563732 7012 server_base.cc:1034] running on GCE node
I20260929 15:52:08.564527 7012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:52:08.565883 7012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:52:08.567118 7012 hybrid_clock.cc:648] HybridClock initialized: now 1790697128567059 us; error 41 us; skew 500 ppm
I20260929 15:52:08.567435 7012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:52:08 dist-test-slave-r0ck krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697128, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:52:08.570389 7012 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260929 15:52:08.571884 7012 webserver.cc:460] Webserver started at http://127.0.97.61:39043/ using document root <none> and password file <none>
I20260929 15:52:08.572531 7012 fs_manager.cc:359] Metadata directory not provided
I20260929 15:52:08.572655 7012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:52:08.572958 7012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:52:08.575704 7012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-1/data/instance:
uuid: "b2ca5a81736e4a508dea4e247fce6a74"
format_stamp: "Formatted at 2026-09-29 15:52:08 on dist-test-slave-r0ck"
server_key: "457ae295332f974001307ac366748b65"
server_key_iv: "4c4c44040788d12c058dd2bb63fadae7"
server_key_version: "encryptionkey@0"
I20260929 15:52:08.576459 7012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-1/wal/instance:
uuid: "b2ca5a81736e4a508dea4e247fce6a74"
format_stamp: "Formatted at 2026-09-29 15:52:08 on dist-test-slave-r0ck"
server_key: "457ae295332f974001307ac366748b65"
server_key_iv: "4c4c44040788d12c058dd2bb63fadae7"
server_key_version: "encryptionkey@0"
I20260929 15:52:08.581518 7012 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 15:52:08.584812 7027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:08.586139 7012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:52:08.586314 7012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-1/wal
uuid: "b2ca5a81736e4a508dea4e247fce6a74"
format_stamp: "Formatted at 2026-09-29 15:52:08 on dist-test-slave-r0ck"
server_key: "457ae295332f974001307ac366748b65"
server_key_iv: "4c4c44040788d12c058dd2bb63fadae7"
server_key_version: "encryptionkey@0"
I20260929 15:52:08.586510 7012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:52:08.600291 7012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:52:08.604427 7012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:52:08.604672 7012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:52:08.621543 7012 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:41069
I20260929 15:52:08.621573 7078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:41069 every 8 connection(s)
I20260929 15:52:08.622753 7012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-1/data/info.pb
I20260929 15:52:08.626222 7079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:52:08.630522 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 7012
I20260929 15:52:08.633699 7079 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:52:08 dist-test-slave-r0ck krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 29 15:52:08 dist-test-slave-r0ck krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697128, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 15:52:08.651958 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:08.637553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40252 (local address 127.0.97.62:33067)
0929 15:52:08.637741 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:08.637747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:08.638519 (+ 772us) server_negotiation.cc:407] Connection header received
0929 15:52:08.638674 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:08.638679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:08.638747 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:08.638845 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:08.639788 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:08.640467 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:08.641882 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:08.642286 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:08.642418 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:08.645632 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:08.645641 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:08.645646 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:08.645681 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:08.648884 (+ 3203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:08.649715 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:08.649719 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:08.649721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:08.649870 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:08.650243 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:08.650248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:08.650252 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:08.650678 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:08.650812 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:08.651441 (+ 629us) server_negotiation.cc:299] Negotiation successful
0929 15:52:08.651644 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 15:52:08.652761 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:08.632619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59358 (local address 127.0.97.61:41069)
0929 15:52:08.633411 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:08.633422 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:08.633460 (+ 38us) server_negotiation.cc:407] Connection header received
0929 15:52:08.634972 (+ 1512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:08.634991 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:08.635348 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:08.635634 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:08.637331 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:08.638936 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:08.640122 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:08.640726 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:08.640915 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:08.645042 (+ 4127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:08.645049 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:08.645059 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:08.645123 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:08.648469 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:08.649191 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:08.649198 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:08.649200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:08.649410 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:08.649848 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:08.649853 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:08.649856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:08.650538 (+ 682us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:08.650814 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:08.651176 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 15:52:08.651602 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":557,"thread_start_us":375,"threads_started":1}
I20260929 15:52:08.652778 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:08.637623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33067 (local address 127.0.0.1:40252)
0929 15:52:08.638377 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:08.638425 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:08.638561 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:08.638889 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:08.638893 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:08.639428 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:08.639642 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:08.639650 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:08.640596 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:08.640603 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:08.641739 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:08.641750 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:08.642487 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:08.642493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:08.642640 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:08.643194 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:08.643213 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:08.645452 (+ 2239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:08.649018 (+ 3566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:08.649028 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:08.649039 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:08.649593 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:08.649984 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:08.649992 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:08.649995 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:08.650118 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:08.650806 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:08.650815 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:08.651041 (+ 226us) client_negotiation.cc:769] Sending connection context
0929 15:52:08.651278 (+ 237us) client_negotiation.cc:240] Negotiation successful
0929 15:52:08.651758 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":140,"thread_start_us":400,"threads_started":1}
I20260929 15:52:08.654531 7079 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41069 } has no permanent_uuid. Determining permanent_uuid...
I20260929 15:52:08.656073 6874 heartbeater.cc:503] Master 127.0.97.62:33067 was elected leader, sending a full tablet report...
Sep 29 15:52:08 dist-test-slave-r0ck krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697128, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260929 15:52:08.666373 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:08.655374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41069 (local address 127.0.0.1:59372)
0929 15:52:08.655560 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:08.655585 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:08.655676 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:08.655943 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:08.655946 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:08.656287 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:08.656490 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:08.656504 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:08.657706 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:08.657710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:08.658551 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:08.658562 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:08.659168 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:08.659171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:08.659298 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:08.659847 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:08.659863 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:08.661730 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:08.664259 (+ 2529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:08.664262 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:08.664265 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:08.664785 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:08.665087 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:08.665091 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:08.665093 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:08.665208 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:08.665710 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:08.665713 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:08.665840 (+ 127us) client_negotiation.cc:769] Sending connection context
0929 15:52:08.665949 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 15:52:08.666133 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 15:52:08.666450 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:08.655413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59372 (local address 127.0.97.61:41069)
0929 15:52:08.655574 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:08.655578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:08.655633 (+ 55us) server_negotiation.cc:407] Connection header received
0929 15:52:08.655760 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:08.655764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:08.655811 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:08.655919 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:08.656608 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:08.657582 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:08.658673 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:08.659051 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:08.659147 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:08.661938 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:08.661943 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:08.661947 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:08.661975 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:08.664150 (+ 2175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:08.664877 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:08.664880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:08.664882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:08.664998 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:08.665293 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:08.665296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:08.665298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:08.665601 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:08.665705 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:08.666032 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 15:52:08.666228 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":66}
I20260929 15:52:08.670928 7079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74: Bootstrap starting.
I20260929 15:52:08.673859 7079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74: Neither blocks nor log segments found. Creating new log.
I20260929 15:52:08.674803 7079 log.cc:824] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74: Log is configured to *not* fsync() on all Append() calls
I20260929 15:52:08.677317 7079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74: No bootstrap required, opened a new log
I20260929 15:52:08.684115 7079 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41069 } }
I20260929 15:52:08.684438 7079 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:52:08.684546 7079 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2ca5a81736e4a508dea4e247fce6a74, State: Initialized, Role: FOLLOWER
I20260929 15:52:08.685092 7079 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41069 } }
I20260929 15:52:08.685905 7085 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41069 } } }
I20260929 15:52:08.686179 7085 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:52:08.686866 7079 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 15:52:08.695165 7096 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 15:52:08.695309 7096 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 15:52:08.697958 7012 master_runner.cc:428] Detected that this master b2ca5a81736e4a508dea4e247fce6a74 is joining an existing cluster
I20260929 15:52:08.698062 7012 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:41069
I20260929 15:52:08.718467 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:08.702567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59380 (local address 127.0.97.61:41069)
0929 15:52:08.702726 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:08.702733 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:08.703268 (+ 535us) server_negotiation.cc:407] Connection header received
0929 15:52:08.703436 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:08.703441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:08.703522 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:08.703627 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:08.704808 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:08.705583 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:08.707656 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:08.708469 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:08.708579 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:08.711686 (+ 3107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:08.711693 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:08.711700 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:08.711749 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:08.715710 (+ 3961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:08.716453 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:08.716461 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:08.716465 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:08.716602 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:08.717008 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:08.717014 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:08.717019 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:08.717397 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:08.717549 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:08.718075 (+ 526us) server_negotiation.cc:299] Negotiation successful
0929 15:52:08.718220 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 15:52:08.718564 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:08.701648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40262 (local address 127.0.97.62:33067)
0929 15:52:08.702440 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:08.702447 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:08.702578 (+ 131us) server_negotiation.cc:407] Connection header received
0929 15:52:08.702882 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:08.702889 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:08.702977 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:08.703054 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:08.704604 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:08.705759 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:08.707882 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:08.708515 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:08.708614 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:08.711403 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:08.711411 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:08.711416 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:08.711453 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:08.714792 (+ 3339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:08.716195 (+ 1403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:08.716201 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:08.716204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:08.716352 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:08.716814 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:08.716820 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:08.716823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:08.717171 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:08.717289 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:08.717971 (+ 682us) server_negotiation.cc:299] Negotiation successful
0929 15:52:08.718191 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662}
I20260929 15:52:08.719563 7103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:08.701974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41069 (local address 127.0.0.1:59380)
0929 15:52:08.703161 (+ 1187us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:08.703179 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:08.703296 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:08.703676 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:08.703680 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:08.704365 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:08.704681 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:08.704689 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:08.705816 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:08.705824 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:08.707476 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:08.707493 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:08.708951 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:08.708964 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:08.709197 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:08.710256 (+ 1059us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:08.710282 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:08.711525 (+ 1243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:08.715846 (+ 4321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:08.715850 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:08.715852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:08.716206 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:08.716710 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:08.716718 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:08.716724 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:08.716890 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:08.717637 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:08.717644 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:08.717816 (+ 172us) client_negotiation.cc:769] Sending connection context
0929 15:52:08.718657 (+ 841us) client_negotiation.cc:240] Negotiation successful
0929 15:52:08.718957 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1064,"mutex_wait_us":27,"thread_start_us":883,"threads_started":1}
I20260929 15:52:08.719602 7102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:08.701803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33067 (local address 127.0.0.1:40262)
0929 15:52:08.702480 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:08.702502 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:08.702699 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:08.703274 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:08.703278 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:08.704032 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:08.704440 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:08.704455 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:08.705937 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:08.705943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:08.707701 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:08.707715 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:08.708906 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:08.708912 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:08.709123 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:08.709885 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:08.709908 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:08.711205 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:08.714982 (+ 3777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:08.714988 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:08.714992 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:08.716062 (+ 1070us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:08.716514 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:08.716519 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:08.716523 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:08.716684 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:08.717415 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:08.717420 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:08.717621 (+ 201us) client_negotiation.cc:769] Sending connection context
0929 15:52:08.718789 (+ 1168us) client_negotiation.cc:240] Negotiation successful
0929 15:52:08.719124 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":36,"thread_start_us":397,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
Sep 29 15:52:08 dist-test-slave-r0ck krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697126, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260929 15:52:08.747073 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:08.726766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59394 (local address 127.0.97.61:41069)
0929 15:52:08.726933 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:08.726939 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:08.729812 (+ 2873us) server_negotiation.cc:407] Connection header received
0929 15:52:08.730219 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:08.730226 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:08.730351 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:08.730437 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:08.732268 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:08.733463 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:08.735322 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:08.735828 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:08.735928 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:08.740136 (+ 4208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:08.740143 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:08.740149 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:08.740184 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:08.743919 (+ 3735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:08.744820 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:08.744825 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:08.744830 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:08.744983 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:08.745437 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:08.745442 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:08.745445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:08.745775 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:08.745936 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:08.746494 (+ 558us) server_negotiation.cc:299] Negotiation successful
0929 15:52:08.746707 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 15:52:08.747669 7105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:08.726413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41069 (local address 127.0.0.1:59394)
0929 15:52:08.729670 (+ 3257us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:08.729712 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:08.729965 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:08.730797 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:08.730808 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:08.731710 (+ 902us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:08.732062 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:08.732078 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:08.733595 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:08.733600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:08.735172 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:08.735187 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:08.735993 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:08.736000 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:08.736197 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:08.736973 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:08.737000 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:08.739927 (+ 2927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:08.744077 (+ 4150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:08.744090 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:08.744101 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:08.744691 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:08.745105 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:08.745113 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:08.745120 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:08.745295 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:08.745922 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:08.745933 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:08.746151 (+ 218us) client_negotiation.cc:769] Sending connection context
0929 15:52:08.746356 (+ 205us) client_negotiation.cc:240] Negotiation successful
0929 15:52:08.746708 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3023,"thread_start_us":2832,"threads_started":1}
I20260929 15:52:08.749739 6551 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:41069: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 41069
}
attrs {
promote: true
}
}
dest_uuid: "e21605ed4e8e4e9189d58164059f7de4"
cas_config_opid_index: -1
I20260929 15:52:08.750924 6551 consensus_queue.cc:237] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: true } }
I20260929 15:52:08.757380 7109 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER b2ca5a81736e4a508dea4e247fce6a74 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: true } } }
I20260929 15:52:08.758517 7054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:52:08.761623 7109 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:41069
I20260929 15:52:08.762408 7012 master.cc:537] Master@127.0.97.61:41069 shutting down...
I20260929 15:52:08.762092 7108 sys_catalog.cc:455] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e21605ed4e8e4e9189d58164059f7de4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: true } } }
I20260929 15:52:08.762915 7108 sys_catalog.cc:458] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [sys.catalog]: This master's current role is: LEADER
I20260929 15:52:08.763545 7054 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [term 1 FOLLOWER]: Refusing update from remote peer e21605ed4e8e4e9189d58164059f7de4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260929 15:52:08.764667 7108 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:52:08.766607 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:36403
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33067,127.0.97.61:41069,127.0.97.60:36403
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:52:08.769676 6535 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260929 15:52:08.773156 6535 consensus_peers.cc:487] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 -> Peer b2ca5a81736e4a508dea4e247fce6a74 (127.0.97.61:41069): Couldn't send request to peer b2ca5a81736e4a508dea4e247fce6a74. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 15:52:08.775408 7012 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:52:08.775718 7012 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:52:08.775856 7012 tablet_replica.cc:330] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74: stopping tablet replica
I20260929 15:52:08.781589 7012 master.cc:552] Master@127.0.97.61:41069 shutdown complete.
I20260929 15:52:08.781669 7012 master_runner.cc:305] Clearing existing system tablet
I20260929 15:52:08.782069 7012 fs_manager.cc:359] Metadata directory not provided
I20260929 15:52:08.782155 7012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:52:08.785033 7012 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 15:52:08.786721 7114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:08.787050 7012 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:52:08.787393 7012 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260929 15:52:08.787510 7012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-1/wal
uuid: "b2ca5a81736e4a508dea4e247fce6a74"
format_stamp: "Formatted at 2026-09-29 15:52:08 on dist-test-slave-r0ck"
server_key: "457ae295332f974001307ac366748b65"
server_key_iv: "4c4c44040788d12c058dd2bb63fadae7"
server_key_version: "encryptionkey@0"
I20260929 15:52:08.792824 7012 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:52:08.799659 7012 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 15:52:08.799861 7012 log.cc:1196] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 15:52:08.800315 7012 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74: Deleting consensus metadata
I20260929 15:52:08.800621 7012 master_runner.cc:315] Copying system tablet from 127.0.97.62:33067
I20260929 15:52:08.806821 7012 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:33067
I20260929 15:52:08.824198 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:08.807545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33067 (local address 127.0.0.1:40276)
0929 15:52:08.809730 (+ 2185us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:08.809755 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:08.809901 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:08.810301 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:08.810307 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:08.810816 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:08.811078 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:08.811091 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:08.812577 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:08.812583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:08.814091 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:08.814110 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:08.815126 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:08.815132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:08.815312 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:08.816024 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:08.816048 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:08.817058 (+ 1010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:08.821274 (+ 4216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:08.821279 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:08.821284 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:08.821771 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:08.822210 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:08.822215 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:08.822219 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:08.822378 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:08.823090 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:08.823095 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:08.823298 (+ 203us) client_negotiation.cc:769] Sending connection context
0929 15:52:08.823440 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 15:52:08.823739 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2035,"thread_start_us":1947,"threads_started":1}
I20260929 15:52:08.824306 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:08.807618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40276 (local address 127.0.97.62:33067)
0929 15:52:08.807810 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:08.807817 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:08.809862 (+ 2045us) server_negotiation.cc:407] Connection header received
0929 15:52:08.810032 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:08.810038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:08.810136 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:08.810248 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:08.811273 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:08.812398 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:08.814249 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:08.814960 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:08.815113 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:08.817278 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:08.817287 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:08.817292 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:08.817329 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:08.821046 (+ 3717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:08.821908 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:08.821916 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:08.821921 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:08.822082 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:08.822528 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:08.822534 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:08.822539 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:08.822955 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:08.823129 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:08.823625 (+ 496us) server_negotiation.cc:299] Negotiation successful
0929 15:52:08.823979 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 15:52:08.825579 6571 tablet_copy_service.cc:140] P e21605ed4e8e4e9189d58164059f7de4: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b2ca5a81736e4a508dea4e247fce6a74 ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:40276)
I20260929 15:52:08.825865 6571 tablet_copy_service.cc:161] P e21605ed4e8e4e9189d58164059f7de4: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b2ca5a81736e4a508dea4e247fce6a74 at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:40276: session id = b2ca5a81736e4a508dea4e247fce6a74-00000000000000000000000000000000
I20260929 15:52:08.830124 6571 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 15:52:08.832346 7012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 15:52:08.838950 7012 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74: tablet copy: Starting download of 0 data blocks...
I20260929 15:52:08.839243 7012 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74: tablet copy: Starting download of 1 WAL segments...
I20260929 15:52:08.842020 7012 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 15:52:08.846421 6571 tablet_copy_service.cc:342] P e21605ed4e8e4e9189d58164059f7de4: Request end of tablet copy session b2ca5a81736e4a508dea4e247fce6a74-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:40276
I20260929 15:52:08.846659 6571 tablet_copy_service.cc:433] P e21605ed4e8e4e9189d58164059f7de4: ending tablet copy session b2ca5a81736e4a508dea4e247fce6a74-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b2ca5a81736e4a508dea4e247fce6a74
I20260929 15:52:08.858982 7012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:52:08.861734 7121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:08.862308 7122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:08.867619 7124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:52:08.870808 7012 server_base.cc:1034] running on GCE node
I20260929 15:52:08.871183 7012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:52:08.871816 7012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:52:08.873039 7012 hybrid_clock.cc:648] HybridClock initialized: now 1790697128872972 us; error 41 us; skew 500 ppm
I20260929 15:52:08.873276 7012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:52:08 dist-test-slave-r0ck krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697128, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:52:08.875499 7012 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260929 15:52:08.876693 7012 webserver.cc:460] Webserver started at http://127.0.97.61:33693/ using document root <none> and password file <none>
I20260929 15:52:08.877064 7012 fs_manager.cc:359] Metadata directory not provided
I20260929 15:52:08.877143 7012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:52:08.880534 7012 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260929 15:52:08.882401 7131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:08.882817 7012 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 15:52:08.882953 7012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-1/wal
uuid: "b2ca5a81736e4a508dea4e247fce6a74"
format_stamp: "Formatted at 2026-09-29 15:52:08 on dist-test-slave-r0ck"
server_key: "457ae295332f974001307ac366748b65"
server_key_iv: "4c4c44040788d12c058dd2bb63fadae7"
server_key_version: "encryptionkey@0"
I20260929 15:52:08.883111 7012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:52:08.910360 7012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:52:08.914948 7012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:52:08.941140 7012 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:41069
I20260929 15:52:08.941166 7182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:41069 every 8 connection(s)
I20260929 15:52:08.942248 7012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-1/data/info.pb
I20260929 15:52:08.947945 7183 sys_catalog.cc:263] Verifying existing consensus state
I20260929 15:52:08.951680 7183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74: Bootstrap starting.
I20260929 15:52:08.967935 7183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 15:52:08.968408 7183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74: Bootstrap complete.
I20260929 15:52:08.969399 7183 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: true } }
I20260929 15:52:08.969632 7183 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b2ca5a81736e4a508dea4e247fce6a74, State: Initialized, Role: LEARNER
I20260929 15:52:08.969854 7183 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: true } }
I20260929 15:52:08.970799 7183 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:52:08.970600 7186 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: true } } }
I20260929 15:52:08.971028 7186 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [sys.catalog]: This master's current role is: LEARNER
I20260929 15:52:08.980926 7197 catalog_manager.cc:1259] Loaded cluster ID: 157dc853f93b4db599b14cf25dcddf85
I20260929 15:52:08.981031 7197 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74: loading cluster ID for follower catalog manager: success
I20260929 15:52:08.984566 7197 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74: acquiring CA information for follower catalog manager: success
I20260929 15:52:08.986790 7197 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 15:52:09.162463 7112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:52:09.162725 7112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20260929 15:52:09.162798 7007 heartbeater.cc:503] Master 127.0.97.62:33067 was elected leader, sending a full tablet report...
W20260929 15:52:09.162881 7112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:52:09.170969 7112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:52:09.171075 7112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:52:09.171165 7112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:52:09.171254 7112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:52:09.171300 7112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:52:09.181205 7112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33067,127.0.97.61:41069,127.0.97.60:36403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:36403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:52:09.182499 7112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:52:09.183905 7112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:52:09.195667 7203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:09.195700 7205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:09.195765 7202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:52:09.196491 7112 server_base.cc:1034] running on GCE node
I20260929 15:52:09.197197 7112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:52:09.198514 7112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:52:09.199699 7112 hybrid_clock.cc:648] HybridClock initialized: now 1790697129199648 us; error 49 us; skew 500 ppm
I20260929 15:52:09.200026 7112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:52:09 dist-test-slave-r0ck krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697129, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:52:09.203014 7112 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260929 15:52:09.204339 7112 webserver.cc:460] Webserver started at http://127.0.97.60:46075/ using document root <none> and password file <none>
I20260929 15:52:09.205138 7112 fs_manager.cc:359] Metadata directory not provided
I20260929 15:52:09.205286 7112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:52:09.205673 7112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:52:09.208811 7112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-2/data/instance:
uuid: "df4d8c2416f947dca2364e0321d98599"
format_stamp: "Formatted at 2026-09-29 15:52:09 on dist-test-slave-r0ck"
server_key: "a33f15cbf23daf3177b6092c36c9004e"
server_key_iv: "6fabe52092717914d04e1438624420ba"
server_key_version: "encryptionkey@0"
I20260929 15:52:09.209580 7112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-2/wal/instance:
uuid: "df4d8c2416f947dca2364e0321d98599"
format_stamp: "Formatted at 2026-09-29 15:52:09 on dist-test-slave-r0ck"
server_key: "a33f15cbf23daf3177b6092c36c9004e"
server_key_iv: "6fabe52092717914d04e1438624420ba"
server_key_version: "encryptionkey@0"
I20260929 15:52:09.214146 7112 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 15:52:09.217257 7212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:09.218531 7112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:52:09.218662 7112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-2/wal
uuid: "df4d8c2416f947dca2364e0321d98599"
format_stamp: "Formatted at 2026-09-29 15:52:09 on dist-test-slave-r0ck"
server_key: "a33f15cbf23daf3177b6092c36c9004e"
server_key_iv: "6fabe52092717914d04e1438624420ba"
server_key_version: "encryptionkey@0"
I20260929 15:52:09.218811 7112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:52:09.237758 7112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:52:09.242072 7112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:52:09.242293 7112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:52:09.260958 7112 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:36403
I20260929 15:52:09.260962 7263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:36403 every 8 connection(s)
I20260929 15:52:09.262322 7112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-2/data/info.pb
I20260929 15:52:09.267007 7264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:52:09.271296 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 7112
I20260929 15:52:09.274461 7264 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:52:09 dist-test-slave-r0ck krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697129, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 29 15:52:09 dist-test-slave-r0ck krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260929 15:52:09.296016 7271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:09.285762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59408 (local address 127.0.97.61:41069)
0929 15:52:09.289200 (+ 3438us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:09.289206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:09.289437 (+ 231us) server_negotiation.cc:407] Connection header received
0929 15:52:09.289614 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:09.289619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:09.289723 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:09.289855 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:52:09.290333 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.291555 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.293442 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.294708 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.294880 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:09.295059 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:09.295553 (+ 494us) server_negotiation.cc:299] Negotiation successful
0929 15:52:09.295687 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3288,"thread_start_us":3190,"threads_started":1}
I20260929 15:52:09.296051 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:09.285577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41069 (local address 127.0.0.1:59408)
0929 15:52:09.289325 (+ 3748us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:09.289363 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:09.289481 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:09.289900 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:09.289907 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:09.289940 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:52:09.290185 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:09.290196 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.291717 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.291723 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:09.293261 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:09.293276 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.295168 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.295177 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:09.295354 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:09.295414 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 15:52:09.295570 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 15:52:09.295697 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3641,"thread_start_us":3568,"threads_started":1}
I20260929 15:52:09.297086 7158 raft_consensus.cc:1203] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260929 15:52:09.297667 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:09.278481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40282 (local address 127.0.97.62:33067)
0929 15:52:09.278633 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:09.278637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:09.279405 (+ 768us) server_negotiation.cc:407] Connection header received
0929 15:52:09.279629 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:09.279633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:09.279693 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:09.279795 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:09.280851 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.281655 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.283304 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.283779 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.283898 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:09.290288 (+ 6390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:09.290296 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:09.290301 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:09.290337 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:09.294065 (+ 3728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:09.294802 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:09.294809 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:09.294812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:09.294933 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:09.295368 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:09.295374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:09.295377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:09.296198 (+ 821us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:09.296322 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:09.297089 (+ 767us) server_negotiation.cc:299] Negotiation successful
0929 15:52:09.297289 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 15:52:09.297883 7186 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e21605ed4e8e4e9189d58164059f7de4. Latest consensus state: current_term: 1 leader_uuid: "e21605ed4e8e4e9189d58164059f7de4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: true } } }
I20260929 15:52:09.298115 7186 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [sys.catalog]: This master's current role is: LEARNER
I20260929 15:52:09.298388 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:09.273407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57990 (local address 127.0.97.60:36403)
0929 15:52:09.274224 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:09.274235 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:09.274273 (+ 38us) server_negotiation.cc:407] Connection header received
0929 15:52:09.275726 (+ 1453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:09.275745 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:09.276122 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:09.276445 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:09.278164 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.280026 (+ 1862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.281309 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.282021 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.282226 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:09.289168 (+ 6942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:09.289178 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:09.289192 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:09.289230 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:09.293095 (+ 3865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:09.293819 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:09.293825 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:09.293831 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:09.293981 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:09.294435 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:09.294440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:09.294443 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:09.295080 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:09.295396 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:09.296599 (+ 1203us) server_negotiation.cc:299] Negotiation successful
0929 15:52:09.297004 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":354,"threads_started":1}
I20260929 15:52:09.298410 7268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:09.278453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33067 (local address 127.0.0.1:40282)
0929 15:52:09.279268 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:09.279309 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:09.279493 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:09.279855 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:09.279861 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:09.280474 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:09.280698 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:09.280706 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.281799 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.281804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:09.283152 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:09.283166 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.284056 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.284063 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:09.284194 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:09.284793 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:09.284814 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:09.290086 (+ 5272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:09.294198 (+ 4112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:09.294210 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:09.294219 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:09.294687 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:09.295044 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:09.295049 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:09.295053 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:09.295224 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:09.296390 (+ 1166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:09.296400 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:09.296596 (+ 196us) client_negotiation.cc:769] Sending connection context
0929 15:52:09.296791 (+ 195us) client_negotiation.cc:240] Negotiation successful
0929 15:52:09.297186 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"mutex_wait_us":76,"thread_start_us":422,"threads_started":1}
I20260929 15:52:09.299988 7264 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41069 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:52:09 dist-test-slave-r0ck krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697129, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260929 15:52:09.311655 7271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:09.300772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59416 (local address 127.0.97.61:41069)
0929 15:52:09.300928 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:09.300935 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:09.300949 (+ 14us) server_negotiation.cc:407] Connection header received
0929 15:52:09.300988 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:09.300992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:09.301061 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:09.301184 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:09.301843 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.302469 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.303648 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.304047 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.304174 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:09.306953 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:09.306959 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:09.306964 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:09.307002 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:09.309264 (+ 2262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:09.309901 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:09.309906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:09.309910 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:09.310041 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:09.310401 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:09.310407 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:09.310411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:09.310652 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:09.310800 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:09.311131 (+ 331us) server_negotiation.cc:299] Negotiation successful
0929 15:52:09.311336 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 15:52:09.311655 7268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:09.300597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41069 (local address 127.0.0.1:59416)
0929 15:52:09.300749 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:09.300767 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:09.300856 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:09.301197 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:09.301201 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:09.301579 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:09.301694 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:09.301700 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.302625 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.302630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:09.303513 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:09.303522 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.304183 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.304188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:09.304399 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:09.304935 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:09.304951 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:09.306789 (+ 1838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:09.309427 (+ 2638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:09.309434 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:09.309437 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:09.309779 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:09.310170 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:09.310174 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:09.310178 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:09.310299 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:09.310767 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:09.310771 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:09.310906 (+ 135us) client_negotiation.cc:769] Sending connection context
0929 15:52:09.311049 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 15:52:09.311306 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 15:52:09.312564 7264 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:52:09 dist-test-slave-r0ck krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697129, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260929 15:52:09.324518 7268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:09.313065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36403 (local address 127.0.0.1:58006)
0929 15:52:09.313205 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:09.313219 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:09.313290 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:09.313657 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:09.313660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:09.313985 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:09.314103 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:09.314110 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.315391 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.315398 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:09.316229 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:09.316239 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.316858 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.316863 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:09.317112 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:09.317636 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:09.317652 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:09.319557 (+ 1905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:09.322464 (+ 2907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:09.322467 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:09.322469 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:09.322799 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:09.323110 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:09.323113 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:09.323115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:09.323264 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:09.323790 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:09.323793 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:09.323904 (+ 111us) client_negotiation.cc:769] Sending connection context
0929 15:52:09.324013 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 15:52:09.324177 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":4}
I20260929 15:52:09.324524 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:09.313287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58006 (local address 127.0.97.60:36403)
0929 15:52:09.313453 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:09.313457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:09.313468 (+ 11us) server_negotiation.cc:407] Connection header received
0929 15:52:09.313504 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:09.313507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:09.313549 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:09.313644 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:09.314224 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.315272 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.316356 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.316727 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.316852 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:09.319715 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:09.319719 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:09.319722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:09.319754 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:09.322322 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:09.322895 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:09.322899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:09.322901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:09.323017 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:09.323343 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:09.323346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:09.323349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:09.323654 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:09.323781 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:09.324097 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 15:52:09.324278 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":79}
I20260929 15:52:09.329172 7264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599: Bootstrap starting.
I20260929 15:52:09.332204 7264 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599: Neither blocks nor log segments found. Creating new log.
I20260929 15:52:09.333148 7264 log.cc:824] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599: Log is configured to *not* fsync() on all Append() calls
I20260929 15:52:09.335990 7264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599: No bootstrap required, opened a new log
I20260929 15:52:09.342657 7264 raft_consensus.cc:348] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41069 } } peers { permanent_uuid: "df4d8c2416f947dca2364e0321d98599" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } }
I20260929 15:52:09.342933 7264 raft_consensus.cc:374] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:52:09.343063 7264 raft_consensus.cc:729] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df4d8c2416f947dca2364e0321d98599, State: Initialized, Role: FOLLOWER
I20260929 15:52:09.343578 7264 consensus_queue.cc:260] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41069 } } peers { permanent_uuid: "df4d8c2416f947dca2364e0321d98599" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } }
I20260929 15:52:09.344383 7273 sys_catalog.cc:455] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41069 } } peers { permanent_uuid: "df4d8c2416f947dca2364e0321d98599" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } } }
I20260929 15:52:09.344717 7273 sys_catalog.cc:458] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:52:09.345317 7264 sys_catalog.cc:564] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:52:09.353300 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 6612
W20260929 15:52:09.355163 7284 catalog_manager.cc:1558] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 15:52:09.355319 7284 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 15:52:09.359812 7112 master_runner.cc:428] Detected that this master df4d8c2416f947dca2364e0321d98599 is joining an existing cluster
I20260929 15:52:09.359925 7112 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:36403
Sep 29 15:52:09 dist-test-slave-r0ck krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260929 15:52:09.375252 7286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:09.354510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46352 (local address 127.0.97.1:43917)
0929 15:52:09.355569 (+ 1059us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:09.355577 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:09.355605 (+ 28us) server_negotiation.cc:407] Connection header received
0929 15:52:09.355665 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:09.355671 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:09.355836 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:09.355994 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:09.357416 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.358971 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.360454 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.361306 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.361511 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:09.367319 (+ 5808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:09.367335 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:09.367349 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:09.367402 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:09.371215 (+ 3813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:09.372016 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:09.372025 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:09.372028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:09.372188 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:09.372636 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:09.372641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:09.372645 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:09.373066 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:09.374431 (+ 1365us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:09.374584 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 15:52:09.374844 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":557,"threads_started":1}
I20260929 15:52:09.376534 6735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:52:09.397786 7292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:09.365057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41069 (local address 127.0.0.1:59422)
0929 15:52:09.372154 (+ 7097us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:09.372187 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:09.372331 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:09.375495 (+ 3164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:09.375503 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:09.376290 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:09.377564 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:09.377584 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.379505 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.379513 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:09.381830 (+ 2317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:09.381848 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.383469 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.383478 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:09.383793 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:09.384639 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:09.384676 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:09.386638 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:09.391967 (+ 5329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:09.391975 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:09.391981 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:09.392616 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:09.393811 (+ 1195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:09.393819 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:09.393826 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:09.394086 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:09.395734 (+ 1648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:09.395742 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:09.396023 (+ 281us) client_negotiation.cc:769] Sending connection context
0929 15:52:09.396207 (+ 184us) client_negotiation.cc:240] Negotiation successful
0929 15:52:09.396644 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6894,"thread_start_us":12006,"threads_started":1}
I20260929 15:52:09.399106 7271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:09.365584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59422 (local address 127.0.97.61:41069)
0929 15:52:09.365789 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:09.365798 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:09.372264 (+ 6466us) server_negotiation.cc:407] Connection header received
0929 15:52:09.372447 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:09.372453 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:09.372527 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:09.372616 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:09.377776 (+ 5160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.379217 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.382062 (+ 2845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.382977 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.383086 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:09.387009 (+ 3923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:09.387018 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:09.387026 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:09.387081 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:09.391751 (+ 4670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:09.392781 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:09.392789 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:09.392795 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:09.392964 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:09.394256 (+ 1292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:09.394264 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:09.394270 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:09.394726 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:09.394894 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:09.398276 (+ 3382us) server_negotiation.cc:299] Negotiation successful
0929 15:52:09.398615 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 15:52:09.409673 7291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:09.365465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33067 (local address 127.0.0.1:40298)
0929 15:52:09.383352 (+ 17887us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:09.383377 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:09.383619 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:09.384322 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:09.384330 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:09.385257 (+ 927us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:09.385736 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:09.385752 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.387657 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.387665 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:09.389715 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:09.389728 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.398889 (+ 9161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.398898 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:09.399165 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:09.400105 (+ 940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:09.400125 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:09.401222 (+ 1097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:09.406482 (+ 5260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:09.406488 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:09.406492 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:09.407012 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:09.407512 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:09.407517 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:09.407521 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:09.407745 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:09.408417 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:09.408422 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:09.408651 (+ 229us) client_negotiation.cc:769] Sending connection context
0929 15:52:09.408796 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 15:52:09.409133 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17671,"thread_start_us":6943,"threads_started":1}
I20260929 15:52:09.410794 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:09.364491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40298 (local address 127.0.97.62:33067)
0929 15:52:09.365081 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:09.365087 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:09.383493 (+ 18406us) server_negotiation.cc:407] Connection header received
0929 15:52:09.383790 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:09.383798 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:09.383906 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:09.383991 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:09.385949 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.387417 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.397552 (+ 10135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.398379 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.398515 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:09.401491 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:09.401502 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:09.401510 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:09.401572 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:09.406282 (+ 4710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:09.407181 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:09.407187 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:09.407191 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:09.407359 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:09.407899 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:09.407906 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:09.407911 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:09.408278 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:09.410061 (+ 1783us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:09.410217 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 15:52:09.410441 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479}
I20260929 15:52:09.424320 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:09.372998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58012 (local address 127.0.97.60:36403)
0929 15:52:09.373229 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:09.373236 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:09.400109 (+ 26873us) server_negotiation.cc:407] Connection header received
0929 15:52:09.400364 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:09.400372 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:09.400464 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:09.400551 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:09.408055 (+ 7504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.409650 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.411756 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.412876 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.413004 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:09.415485 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:09.415492 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:09.415497 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:09.415530 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:09.419098 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:09.422203 (+ 3105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:09.422208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:09.422212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:09.422335 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:09.422732 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:09.422737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:09.422740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:09.423102 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:09.423247 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:09.423747 (+ 500us) server_negotiation.cc:299] Negotiation successful
0929 15:52:09.423962 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 15:52:09.425783 7293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:09.373367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36403 (local address 127.0.0.1:58012)
0929 15:52:09.399971 (+ 26604us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:09.400005 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:09.400215 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:09.400776 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:09.400783 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:09.406634 (+ 5851us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:09.407090 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:09.407107 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.409888 (+ 2781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.409892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:09.411615 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:09.411628 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.413098 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.413105 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:09.413279 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:09.414204 (+ 925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:09.414226 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:09.415309 (+ 1083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:09.421542 (+ 6233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:09.421548 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:09.421553 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:09.422086 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:09.422483 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:09.422488 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:09.422491 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:09.422628 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:09.423311 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:09.423316 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:09.423500 (+ 184us) client_negotiation.cc:769] Sending connection context
0929 15:52:09.424980 (+ 1480us) client_negotiation.cc:240] Negotiation successful
0929 15:52:09.425250 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26429,"thread_start_us":26262,"threads_started":1}
Sep 29 15:52:09 dist-test-slave-r0ck krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697126, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260929 15:52:09.438433 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:09.416823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36403 (local address 127.0.0.1:58016)
0929 15:52:09.416977 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:09.416996 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:09.417095 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:09.424906 (+ 7811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:09.424914 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:09.425561 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:09.425809 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:09.425825 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.428045 (+ 2220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.428051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:09.429335 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:09.429372 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.430108 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.430114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:09.430278 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:09.430871 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:09.430893 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:09.433108 (+ 2215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:09.436083 (+ 2975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:09.436088 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:09.436092 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:09.436450 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:09.436816 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:09.436821 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:09.436825 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:09.436958 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:09.437511 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:09.437515 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:09.437687 (+ 172us) client_negotiation.cc:769] Sending connection context
0929 15:52:09.437829 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 15:52:09.438092 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 15:52:09.438457 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:09.417171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58016 (local address 127.0.97.60:36403)
0929 15:52:09.424545 (+ 7374us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:09.424550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:09.424568 (+ 18us) server_negotiation.cc:407] Connection header received
0929 15:52:09.424657 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:09.424663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:09.424743 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:09.424842 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:09.425994 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.427860 (+ 1866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.429528 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.429952 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.430098 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:09.433285 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:09.433290 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:09.433293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:09.433319 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:09.435902 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:09.436583 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:09.436588 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:09.436591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:09.436704 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:09.437076 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:09.437080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:09.437084 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:09.437331 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:09.437544 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:09.437911 (+ 367us) server_negotiation.cc:299] Negotiation successful
0929 15:52:09.438107 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7264,"thread_start_us":10745,"threads_started":1}
I20260929 15:52:09.439970 6551 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:36403: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "df4d8c2416f947dca2364e0321d98599"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 36403
}
attrs {
promote: true
}
}
dest_uuid: "e21605ed4e8e4e9189d58164059f7de4"
cas_config_opid_index: 5
I20260929 15:52:09.440814 6551 consensus_queue.cc:237] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: true } } peers { permanent_uuid: "df4d8c2416f947dca2364e0321d98599" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attrs { promote: true } }
I20260929 15:52:09.442555 7269 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER df4d8c2416f947dca2364e0321d98599 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: true } } peers { permanent_uuid: "df4d8c2416f947dca2364e0321d98599" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attrs { promote: true } } }
I20260929 15:52:09.443148 7158 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [term 1 LEARNER]: Refusing update from remote peer e21605ed4e8e4e9189d58164059f7de4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 15:52:09.446928 7269 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:36403
I20260929 15:52:09.456067 7297 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 15:52:09.458568 7296 sys_catalog.cc:455] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e21605ed4e8e4e9189d58164059f7de4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: true } } peers { permanent_uuid: "df4d8c2416f947dca2364e0321d98599" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attrs { promote: true } } }
I20260929 15:52:09.458899 7296 sys_catalog.cc:458] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [sys.catalog]: This master's current role is: LEADER
I20260929 15:52:09.469400 7158 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER df4d8c2416f947dca2364e0321d98599 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: true } } peers { permanent_uuid: "df4d8c2416f947dca2364e0321d98599" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attrs { promote: true } } }
I20260929 15:52:09.470228 7239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:52:09.473462 7112 master.cc:537] Master@127.0.97.60:36403 shutting down...
I20260929 15:52:09.474862 7186 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e21605ed4e8e4e9189d58164059f7de4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: true } } peers { permanent_uuid: "df4d8c2416f947dca2364e0321d98599" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attrs { promote: true } } }
I20260929 15:52:09.475154 7186 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [sys.catalog]: This master's current role is: LEARNER
I20260929 15:52:09.476274 7297 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4: attempting to promote NON_VOTER b2ca5a81736e4a508dea4e247fce6a74 to VOTER
I20260929 15:52:09.477464 7297 consensus_queue.cc:237] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: false } } peers { permanent_uuid: "df4d8c2416f947dca2364e0321d98599" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attrs { promote: true } }
I20260929 15:52:09.483373 7158 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [term 1 LEARNER]: Refusing update from remote peer e21605ed4e8e4e9189d58164059f7de4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 15:52:09.484072 7269 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 15:52:09.484424 7239 raft_consensus.cc:1261] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599 [term 1 FOLLOWER]: Refusing update from remote peer e21605ed4e8e4e9189d58164059f7de4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 15:52:09.486835 7269 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b2ca5a81736e4a508dea4e247fce6a74 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: false } } peers { permanent_uuid: "df4d8c2416f947dca2364e0321d98599" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attrs { promote: true } } }
I20260929 15:52:09.491788 7157 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b2ca5a81736e4a508dea4e247fce6a74 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: false } } peers { permanent_uuid: "df4d8c2416f947dca2364e0321d98599" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attrs { promote: true } } }
I20260929 15:52:09.493649 7112 raft_consensus.cc:2229] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:52:09.494027 7112 raft_consensus.cc:2258] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:52:09.494227 7112 tablet_replica.cc:330] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599: stopping tablet replica
I20260929 15:52:09.496243 7186 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e21605ed4e8e4e9189d58164059f7de4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: false } } peers { permanent_uuid: "df4d8c2416f947dca2364e0321d98599" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attrs { promote: true } } }
I20260929 15:52:09.496465 7186 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:52:09.505788 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:09.505038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36403 (local address 127.0.0.1:58024)
0929 15:52:09.505229 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:09.505522 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 15:52:09.506124 6538 consensus_peers.cc:487] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 -> Peer df4d8c2416f947dca2364e0321d98599 (127.0.97.60:36403): Couldn't send request to peer df4d8c2416f947dca2364e0321d98599. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36403: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 15:52:09.507411 7112 master.cc:552] Master@127.0.97.60:36403 shutdown complete.
I20260929 15:52:09.507516 7112 master_runner.cc:305] Clearing existing system tablet
I20260929 15:52:09.507992 7112 fs_manager.cc:359] Metadata directory not provided
I20260929 15:52:09.508121 7112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:52:09.512028 7302 sys_catalog.cc:455] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e21605ed4e8e4e9189d58164059f7de4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: false } } peers { permanent_uuid: "df4d8c2416f947dca2364e0321d98599" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attrs { promote: true } } }
I20260929 15:52:09.512374 7302 sys_catalog.cc:458] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [sys.catalog]: This master's current role is: LEADER
I20260929 15:52:09.512831 7296 sys_catalog.cc:455] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e21605ed4e8e4e9189d58164059f7de4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: false } } peers { permanent_uuid: "df4d8c2416f947dca2364e0321d98599" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attrs { promote: true } } }
I20260929 15:52:09.513082 7296 sys_catalog.cc:458] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [sys.catalog]: This master's current role is: LEADER
I20260929 15:52:09.513963 7112 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 15:52:09.515431 7304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:09.515753 7112 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:52:09.516055 7112 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260929 15:52:09.516175 7112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-2/wal
uuid: "df4d8c2416f947dca2364e0321d98599"
format_stamp: "Formatted at 2026-09-29 15:52:09 on dist-test-slave-r0ck"
server_key: "a33f15cbf23daf3177b6092c36c9004e"
server_key_iv: "6fabe52092717914d04e1438624420ba"
server_key_version: "encryptionkey@0"
I20260929 15:52:09.519487 7112 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:52:09.524221 7112 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 15:52:09.524384 7112 log.cc:1196] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 15:52:09.524801 7112 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599: Deleting consensus metadata
I20260929 15:52:09.525064 7112 master_runner.cc:315] Copying system tablet from 127.0.97.62:33067
I20260929 15:52:09.527951 7112 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:33067
I20260929 15:52:09.540225 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:09.528690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40314 (local address 127.0.97.62:33067)
0929 15:52:09.528947 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:09.528954 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:09.529416 (+ 462us) server_negotiation.cc:407] Connection header received
0929 15:52:09.529644 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:09.529649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:09.529725 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:09.529829 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:09.530614 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.531577 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.532856 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.533268 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.533438 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:09.535153 (+ 1715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:09.535159 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:09.535163 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:09.535202 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:09.537726 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:09.538334 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:09.538341 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:09.538345 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:09.538469 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:09.538838 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:09.538845 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:09.538851 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:09.539107 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:09.539257 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:09.539641 (+ 384us) server_negotiation.cc:299] Negotiation successful
0929 15:52:09.539859 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260929 15:52:09.540249 7310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:09.528676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33067 (local address 127.0.0.1:40314)
0929 15:52:09.529289 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:09.529305 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:09.529515 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:09.529850 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:09.529855 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:09.530268 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:09.530456 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:09.530464 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.531733 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.531739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:09.532719 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:09.532730 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.533441 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.533446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:09.533630 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:09.534275 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:09.534293 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:09.534951 (+ 658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:09.537858 (+ 2907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:09.537864 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:09.537868 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:09.538212 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:09.538582 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:09.538587 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:09.538591 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:09.538731 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:09.539225 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:09.539230 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:09.539378 (+ 148us) client_negotiation.cc:769] Sending connection context
0929 15:52:09.539530 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 15:52:09.539818 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":370,"threads_started":1}
I20260929 15:52:09.541047 6571 tablet_copy_service.cc:140] P e21605ed4e8e4e9189d58164059f7de4: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer df4d8c2416f947dca2364e0321d98599 ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:40314)
I20260929 15:52:09.541208 6571 tablet_copy_service.cc:161] P e21605ed4e8e4e9189d58164059f7de4: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer df4d8c2416f947dca2364e0321d98599 at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:40314: session id = df4d8c2416f947dca2364e0321d98599-00000000000000000000000000000000
I20260929 15:52:09.543419 6571 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 15:52:09.544837 7112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 15:52:09.550401 7112 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599: tablet copy: Starting download of 0 data blocks...
I20260929 15:52:09.550669 7112 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599: tablet copy: Starting download of 1 WAL segments...
I20260929 15:52:09.552640 7112 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 15:52:09.555627 6571 tablet_copy_service.cc:342] P e21605ed4e8e4e9189d58164059f7de4: Request end of tablet copy session df4d8c2416f947dca2364e0321d98599-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:40314
I20260929 15:52:09.555754 6571 tablet_copy_service.cc:433] P e21605ed4e8e4e9189d58164059f7de4: ending tablet copy session df4d8c2416f947dca2364e0321d98599-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer df4d8c2416f947dca2364e0321d98599
I20260929 15:52:09.560009 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 6612
I20260929 15:52:09.567651 7112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:52:09.570607 7311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:09.578994 7312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:52:09.579883 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:43917
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:33067
--webserver_port=43383
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:33067,127.0.97.61:41069
--tserver_master_addrs=127.0.97.62:33067,127.0.97.61:41069,127.0.97.60:36403 with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:52:09.580147 7314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:52:09.587143 7112 server_base.cc:1034] running on GCE node
I20260929 15:52:09.587473 7112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:52:09.599637 7112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:52:09.602974 7112 hybrid_clock.cc:648] HybridClock initialized: now 1790697129602927 us; error 44 us; skew 500 ppm
I20260929 15:52:09.603293 7112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:52:09 dist-test-slave-r0ck krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697129, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:52:09.605562 7112 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260929 15:52:09.618105 7112 webserver.cc:460] Webserver started at http://127.0.97.60:37253/ using document root <none> and password file <none>
I20260929 15:52:09.618450 7112 fs_manager.cc:359] Metadata directory not provided
I20260929 15:52:09.618561 7112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:52:09.622013 7112 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260929 15:52:09.624003 7322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:09.624444 7112 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20260929 15:52:09.624601 7112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-2/wal
uuid: "df4d8c2416f947dca2364e0321d98599"
format_stamp: "Formatted at 2026-09-29 15:52:09 on dist-test-slave-r0ck"
server_key: "a33f15cbf23daf3177b6092c36c9004e"
server_key_iv: "6fabe52092717914d04e1438624420ba"
server_key_version: "encryptionkey@0"
I20260929 15:52:09.624827 7112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:52:09.658066 7112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:52:09.662205 7112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:52:09.704698 7112 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:36403
I20260929 15:52:09.704763 7373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:36403 every 8 connection(s)
I20260929 15:52:09.705799 7112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/master-2/data/info.pb
I20260929 15:52:09.711019 7374 sys_catalog.cc:263] Verifying existing consensus state
I20260929 15:52:09.714560 7374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599: Bootstrap starting.
I20260929 15:52:09.737113 7374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 15:52:09.737653 7374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599: Bootstrap complete.
I20260929 15:52:09.738785 7374 raft_consensus.cc:348] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: false } } peers { permanent_uuid: "df4d8c2416f947dca2364e0321d98599" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attrs { promote: true } }
I20260929 15:52:09.739066 7374 raft_consensus.cc:729] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: df4d8c2416f947dca2364e0321d98599, State: Initialized, Role: LEARNER
I20260929 15:52:09.739317 7374 consensus_queue.cc:260] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: false } } peers { permanent_uuid: "df4d8c2416f947dca2364e0321d98599" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attrs { promote: true } }
I20260929 15:52:09.740154 7377 sys_catalog.cc:455] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: false } } peers { permanent_uuid: "df4d8c2416f947dca2364e0321d98599" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attrs { promote: true } } }
I20260929 15:52:09.740479 7374 sys_catalog.cc:564] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:52:09.740710 7377 sys_catalog.cc:458] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599 [sys.catalog]: This master's current role is: LEARNER
I20260929 15:52:09.764022 7388 catalog_manager.cc:1259] Loaded cluster ID: 157dc853f93b4db599b14cf25dcddf85
I20260929 15:52:09.764127 7388 catalog_manager.cc:1552] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599: loading cluster ID for follower catalog manager: success
I20260929 15:52:09.767652 7388 catalog_manager.cc:1574] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599: acquiring CA information for follower catalog manager: success
I20260929 15:52:09.770020 7388 catalog_manager.cc:1602] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 15:52:09.932723 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:09.925176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36403 (local address 127.0.0.1:58040)
0929 15:52:09.925787 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:09.925813 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:09.925981 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:09.926742 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:09.926747 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:09.926783 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:52:09.926986 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:09.926998 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.928401 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.928409 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:09.930590 (+ 2181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:09.930608 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.931851 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.931859 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:09.932083 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:09.932120 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 15:52:09.932261 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 15:52:09.932420 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 15:52:09.932747 7389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:09.925811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58040 (local address 127.0.97.60:36403)
0929 15:52:09.926392 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:09.926396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:09.926413 (+ 17us) server_negotiation.cc:407] Connection header received
0929 15:52:09.926496 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:09.926503 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:09.926584 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:09.926682 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:52:09.927130 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.928214 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.930779 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:09.931602 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:09.931763 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:09.932027 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:09.932283 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 15:52:09.932420 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":346,"threads_started":1}
I20260929 15:52:09.934005 7349 raft_consensus.cc:1203] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260929 15:52:09.934669 7377 sys_catalog.cc:455] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e21605ed4e8e4e9189d58164059f7de4. Latest consensus state: current_term: 1 leader_uuid: "e21605ed4e8e4e9189d58164059f7de4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: false } } peers { permanent_uuid: "df4d8c2416f947dca2364e0321d98599" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attrs { promote: true } } }
I20260929 15:52:09.934989 7377 sys_catalog.cc:458] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599 [sys.catalog]: This master's current role is: LEARNER
W20260929 15:52:09.962361 7316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:52:09.962620 7316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:52:09.962713 7316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:52:09.970538 7316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:52:09.970665 7316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:52:09.970822 7316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:52:09.980911 7316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:43917
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=43383
--tserver_master_addrs=127.0.97.62:33067,127.0.97.61:41069,127.0.97.60:36403
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:52:09.982255 7316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:52:09.983718 7316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:52:09.997394 7397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:09.997298 7394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:09.997594 7395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:52:09.998425 7316 server_base.cc:1034] running on GCE node
I20260929 15:52:09.998987 7316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:52:09.999882 7316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:52:10.001123 7316 hybrid_clock.cc:648] HybridClock initialized: now 1790697130001073 us; error 35 us; skew 500 ppm
I20260929 15:52:10.001523 7316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:52:09 dist-test-slave-r0ck krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697129, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:52:10.005167 7316 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260929 15:52:10.007025 7316 webserver.cc:460] Webserver started at http://127.0.97.1:43383/ using document root <none> and password file <none>
I20260929 15:52:10.007789 7316 fs_manager.cc:359] Metadata directory not provided
I20260929 15:52:10.007916 7316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:52:10.014078 7316 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 15:52:10.016945 7404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:10.018474 7316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 15:52:10.018657 7316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-0/wal
uuid: "3ab5024e8d1a4d259eefb3f421e119fc"
format_stamp: "Formatted at 2026-09-29 15:52:07 on dist-test-slave-r0ck"
server_key: "bd7b0be039b5009e9e37a85e03f0d97e"
server_key_iv: "def5095a066d96cf95a2a94b2741a749"
server_key_version: "encryptionkey@0"
I20260929 15:52:10.019652 7316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:52:10.040849 7316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:52:10.044941 7316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:52:10.045240 7316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:52:10.046228 7316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:52:10.047561 7316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:52:10.047681 7316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:10.047814 7316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:52:10.047904 7316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:10.082823 7316 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43917
I20260929 15:52:10.082887 7516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43917 every 8 connection(s)
I20260929 15:52:10.084322 7316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:52:10.085939 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 7316
I20260929 15:52:10.086270 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 6746
Sep 29 15:52:10 dist-test-slave-r0ck krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697129, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 29 15:52:10 dist-test-slave-r0ck krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260929 15:52:10.122633 7389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:10.088750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36055 (local address 127.0.97.60:36403)
0929 15:52:10.088995 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:10.089002 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:10.089356 (+ 354us) server_negotiation.cc:407] Connection header received
0929 15:52:10.093302 (+ 3946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:10.093311 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:10.093447 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:10.093550 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:10.096369 (+ 2819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:10.097895 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:10.101487 (+ 3592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:10.102391 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:10.102527 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:10.108766 (+ 6239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:10.108777 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:10.108785 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:10.108846 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:10.113152 (+ 4306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:10.114432 (+ 1280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:10.114441 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:10.114447 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:10.114671 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:10.116772 (+ 2101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:10.116781 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:10.116787 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:10.117243 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:10.117435 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:10.121460 (+ 4025us) server_negotiation.cc:299] Negotiation successful
0929 15:52:10.122104 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 15:52:10.125737 7521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:10.087591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48186 (local address 127.0.97.2:42699)
0929 15:52:10.098551 (+ 10960us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:10.098563 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:10.098599 (+ 36us) server_negotiation.cc:407] Connection header received
0929 15:52:10.098678 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:10.098689 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:10.098968 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:10.102023 (+ 3055us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:10.102314 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:10.104133 (+ 1819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:10.106970 (+ 2837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:10.108796 (+ 1826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:10.110186 (+ 1390us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:10.117111 (+ 6925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:10.117132 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:10.117157 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:10.117217 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:10.121949 (+ 4732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:10.122809 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:10.122817 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:10.122820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:10.122958 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:10.123408 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:10.123413 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:10.123416 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:10.123893 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:10.124752 (+ 859us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:10.125003 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 15:52:10.125291 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10729,"thread_start_us":11519,"threads_started":1}
I20260929 15:52:10.124475 7522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:10.088052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36403 (local address 127.0.97.1:36055)
0929 15:52:10.089147 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:10.089214 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:10.092911 (+ 3697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:10.094186 (+ 1275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:10.094201 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:10.095126 (+ 925us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:10.096106 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:10.096138 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:10.098497 (+ 2359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:10.098506 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:10.101228 (+ 2722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:10.101249 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:10.102955 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:10.102961 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:10.103370 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:10.104263 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:10.104304 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:10.108382 (+ 4078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:10.113422 (+ 5040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:10.113435 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:10.113460 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:10.114250 (+ 790us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:10.114851 (+ 601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:10.114858 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:10.114865 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:10.115170 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:10.120221 (+ 5051us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:10.120234 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:10.120845 (+ 611us) client_negotiation.cc:769] Sending connection context
0929 15:52:10.121936 (+ 1091us) client_negotiation.cc:240] Negotiation successful
0929 15:52:10.122495 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"thread_start_us":819,"threads_started":1}
I20260929 15:52:10.127068 6869 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:52:10.127662 7517 heartbeater.cc:347] Connected to a master server at 127.0.97.60:36403
I20260929 15:52:10.128006 7517 heartbeater.cc:464] Registering TS with master...
I20260929 15:52:10.128958 7517 heartbeater.cc:511] Master 127.0.97.60:36403 requested a full tablet report, sending...
Sep 29 15:52:10 dist-test-slave-r0ck krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697129, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 15:52:10.131457 7339 ts_manager.cc:194] Registered new tserver with Master: 3ab5024e8d1a4d259eefb3f421e119fc (127.0.97.1:43917)
I20260929 15:52:10.139046 7523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:10.090532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45523 (local address 127.0.97.62:33067)
0929 15:52:10.111071 (+ 20539us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:10.111081 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:10.111143 (+ 62us) server_negotiation.cc:407] Connection header received
0929 15:52:10.115316 (+ 4173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:10.115326 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:10.115442 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:10.115586 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:10.118700 (+ 3114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:10.120024 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:10.125526 (+ 5502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:10.126625 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:10.127373 (+ 748us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:10.131764 (+ 4391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:10.131774 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:10.131782 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:10.131834 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:10.136295 (+ 4461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:10.137090 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:10.137094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:10.137096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:10.137207 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:10.137657 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:10.137662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:10.137666 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:10.137928 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:10.138074 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:10.138452 (+ 378us) server_negotiation.cc:299] Negotiation successful
0929 15:52:10.138662 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19408,"thread_start_us":19444,"threads_started":1}
Sep 29 15:52:10 dist-test-slave-r0ck krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697129, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260929 15:52:10.139616 7524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:10.090825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33067 (local address 127.0.97.1:45523)
0929 15:52:10.106551 (+ 15726us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:10.106577 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:10.106717 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:10.115826 (+ 9109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:10.115833 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:10.116636 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:10.117909 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:10.117925 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:10.123415 (+ 5490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:10.123423 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:10.125031 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:10.125046 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:10.126821 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:10.126828 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:10.127012 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:10.127715 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:10.127792 (+ 77us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:10.131368 (+ 3576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:10.136431 (+ 5063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:10.136437 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:10.136441 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:10.136954 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:10.137374 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:10.137378 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:10.137381 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:10.137539 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:10.138059 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:10.138064 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:10.138204 (+ 140us) client_negotiation.cc:769] Sending connection context
0929 15:52:10.138347 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 15:52:10.139330 (+ 983us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15529,"thread_start_us":13278,"threads_started":1}
I20260929 15:52:10.140578 7518 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33067
I20260929 15:52:10.140712 7518 heartbeater.cc:464] Registering TS with master...
I20260929 15:52:10.141199 7518 heartbeater.cc:511] Master 127.0.97.62:33067 requested a full tablet report, sending...
I20260929 15:52:10.142542 6551 ts_manager.cc:194] Re-registered known tserver with Master: 3ab5024e8d1a4d259eefb3f421e119fc (127.0.97.1:43917)
I20260929 15:52:10.143668 6551 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:45523
I20260929 15:52:10.147243 7527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:10.110658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55691 (local address 127.0.97.61:41069)
0929 15:52:10.119811 (+ 9153us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:10.119817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:10.125688 (+ 5871us) server_negotiation.cc:407] Connection header received
0929 15:52:10.125772 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:10.125778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:10.127949 (+ 2171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:10.128089 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:10.130375 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:10.131523 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:10.134501 (+ 2978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:10.135277 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:10.135391 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:10.140333 (+ 4942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:10.140340 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:10.140345 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:10.140383 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:10.143878 (+ 3495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:10.144913 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:10.144920 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:10.144925 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:10.145114 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:10.145661 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:10.145667 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:10.145672 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:10.146084 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:10.146233 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:10.146840 (+ 607us) server_negotiation.cc:299] Negotiation successful
0929 15:52:10.146997 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9035,"thread_start_us":8936,"threads_started":1}
I20260929 15:52:10.147384 7526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:10.110272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41069 (local address 127.0.97.1:55691)
0929 15:52:10.125434 (+ 15162us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:10.125454 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:10.125567 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:10.129036 (+ 3469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:10.129044 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:10.129875 (+ 831us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:10.130162 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:10.130179 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:10.131738 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:10.131744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:10.134319 (+ 2575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:10.134336 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:10.135686 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:10.135695 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:10.135885 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:10.137147 (+ 1262us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:10.137167 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:10.140050 (+ 2883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:10.144178 (+ 4128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:10.144182 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:10.144185 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:10.144752 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:10.145288 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:10.145294 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:10.145299 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:10.145515 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:10.146327 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:10.146334 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:10.146520 (+ 186us) client_negotiation.cc:769] Sending connection context
0929 15:52:10.146677 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 15:52:10.146982 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14970,"spinlock_wait_cycles":21760,"thread_start_us":14952,"threads_started":1}
I20260929 15:52:10.148228 7519 heartbeater.cc:347] Connected to a master server at 127.0.97.61:41069
I20260929 15:52:10.148386 7519 heartbeater.cc:464] Registering TS with master...
I20260929 15:52:10.148747 7519 heartbeater.cc:511] Master 127.0.97.61:41069 requested a full tablet report, sending...
I20260929 15:52:10.150301 7148 ts_manager.cc:194] Registered new tserver with Master: 3ab5024e8d1a4d259eefb3f421e119fc (127.0.97.1:43917)
I20260929 15:52:10.218677 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 6746
I20260929 15:52:10.235457 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:42699
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:33067
--webserver_port=44537
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:33067,127.0.97.61:41069
--tserver_master_addrs=127.0.97.62:33067,127.0.97.61:41069,127.0.97.60:36403 with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:52:10.427819 7530 sys_catalog.cc:455] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e21605ed4e8e4e9189d58164059f7de4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: false } } peers { permanent_uuid: "df4d8c2416f947dca2364e0321d98599" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attrs { promote: true } } }
I20260929 15:52:10.428218 7530 sys_catalog.cc:458] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [sys.catalog]: This master's current role is: LEADER
I20260929 15:52:10.429453 7530 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4: attempting to promote NON_VOTER df4d8c2416f947dca2364e0321d98599 to VOTER
I20260929 15:52:10.430248 7530 consensus_queue.cc:237] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: false } } peers { permanent_uuid: "df4d8c2416f947dca2364e0321d98599" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attrs { promote: false } }
I20260929 15:52:10.432420 7349 raft_consensus.cc:1261] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599 [term 1 LEARNER]: Refusing update from remote peer e21605ed4e8e4e9189d58164059f7de4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 15:52:10.432646 7158 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [term 1 FOLLOWER]: Refusing update from remote peer e21605ed4e8e4e9189d58164059f7de4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 15:52:10.433082 7532 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df4d8c2416f947dca2364e0321d98599" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 15:52:10.433485 7530 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 15:52:10.436383 7532 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, df4d8c2416f947dca2364e0321d98599 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: false } } peers { permanent_uuid: "df4d8c2416f947dca2364e0321d98599" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attrs { promote: false } } }
I20260929 15:52:10.437109 7158 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, df4d8c2416f947dca2364e0321d98599 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: false } } peers { permanent_uuid: "df4d8c2416f947dca2364e0321d98599" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attrs { promote: false } } }
I20260929 15:52:10.440945 7296 sys_catalog.cc:455] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e21605ed4e8e4e9189d58164059f7de4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: false } } peers { permanent_uuid: "df4d8c2416f947dca2364e0321d98599" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attrs { promote: false } } }
I20260929 15:52:10.441232 7296 sys_catalog.cc:458] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [sys.catalog]: This master's current role is: LEADER
I20260929 15:52:10.441478 7296 sys_catalog.cc:455] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e21605ed4e8e4e9189d58164059f7de4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: false } } peers { permanent_uuid: "df4d8c2416f947dca2364e0321d98599" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attrs { promote: false } } }
I20260929 15:52:10.441776 7296 sys_catalog.cc:458] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [sys.catalog]: This master's current role is: LEADER
I20260929 15:52:10.444370 7348 raft_consensus.cc:2932] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, df4d8c2416f947dca2364e0321d98599 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: false } } peers { permanent_uuid: "df4d8c2416f947dca2364e0321d98599" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attrs { promote: false } } }
I20260929 15:52:10.450946 7535 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e21605ed4e8e4e9189d58164059f7de4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: false } } peers { permanent_uuid: "df4d8c2416f947dca2364e0321d98599" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attrs { promote: false } } }
I20260929 15:52:10.451510 7535 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:52:10.452462 7532 sys_catalog.cc:455] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e21605ed4e8e4e9189d58164059f7de4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: false } } peers { permanent_uuid: "df4d8c2416f947dca2364e0321d98599" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attrs { promote: false } } }
I20260929 15:52:10.452841 7532 sys_catalog.cc:458] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4 [sys.catalog]: This master's current role is: LEADER
I20260929 15:52:10.452939 7536 sys_catalog.cc:455] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e21605ed4e8e4e9189d58164059f7de4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: false } } peers { permanent_uuid: "df4d8c2416f947dca2364e0321d98599" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attrs { promote: false } } }
I20260929 15:52:10.453302 7536 sys_catalog.cc:458] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 15:52:10.568375 7529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:52:10.568632 7529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:52:10.568724 7529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:52:10.576257 7529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:52:10.576359 7529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:52:10.576480 7529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:52:10.589354 7529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:42699
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=44537
--tserver_master_addrs=127.0.97.62:33067,127.0.97.61:41069,127.0.97.60:36403
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:52:10.590896 7529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:52:10.592464 7529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:52:10.605541 7542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:10.605578 7544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:10.605515 7541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:52:10.606434 7529 server_base.cc:1034] running on GCE node
I20260929 15:52:10.607110 7529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:52:10.608033 7529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:52:10.609249 7529 hybrid_clock.cc:648] HybridClock initialized: now 1790697130609207 us; error 69 us; skew 500 ppm
I20260929 15:52:10.609591 7529 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:52:10 dist-test-slave-r0ck krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697130, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:52:10.612838 7529 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260929 15:52:10.614405 7529 webserver.cc:460] Webserver started at http://127.0.97.2:44537/ using document root <none> and password file <none>
I20260929 15:52:10.615124 7529 fs_manager.cc:359] Metadata directory not provided
I20260929 15:52:10.615273 7529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:52:10.620394 7529 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260929 15:52:10.623070 7551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:10.624478 7529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 15:52:10.624604 7529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-1/wal
uuid: "2f0aa7392be545f89274d4a92743c022"
format_stamp: "Formatted at 2026-09-29 15:52:07 on dist-test-slave-r0ck"
server_key: "4f91533c312bbd4195c9ed1f9955e2f5"
server_key_iv: "1665efa9fdc040066736d1de10e14c72"
server_key_version: "encryptionkey@0"
I20260929 15:52:10.625638 7529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:52:10.638947 7529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:52:10.643684 7529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:52:10.643954 7529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:52:10.644799 7529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:52:10.646101 7529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:52:10.646209 7529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:10.646337 7529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:52:10.646431 7529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:10.677489 7529 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42699
I20260929 15:52:10.677573 7663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42699 every 8 connection(s)
I20260929 15:52:10.678871 7529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:52:10.685947 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 7529
I20260929 15:52:10.686209 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 6879
Sep 29 15:52:10 dist-test-slave-r0ck krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697130, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 29 15:52:10 dist-test-slave-r0ck krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697130, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 29 15:52:10 dist-test-slave-r0ck krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
Sep 29 15:52:10 dist-test-slave-r0ck krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697130, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260929 15:52:10.720817 7670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:10.682308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36403 (local address 127.0.97.2:43621)
0929 15:52:10.683647 (+ 1339us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:10.683700 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:10.686480 (+ 2780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:10.702973 (+ 16493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:10.702978 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:10.703374 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:10.703622 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:10.703632 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:10.705305 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:10.705311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:10.706942 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:10.706964 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:10.708023 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:10.708030 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:10.708302 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:10.708944 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:10.708984 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:10.712005 (+ 3021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:10.716266 (+ 4261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:10.716275 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:10.716292 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:10.716861 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:10.717325 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:10.717329 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:10.717334 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:10.717630 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:10.718272 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:10.718281 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:10.718690 (+ 409us) client_negotiation.cc:769] Sending connection context
0929 15:52:10.718973 (+ 283us) client_negotiation.cc:240] Negotiation successful
0929 15:52:10.719468 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":976,"mutex_wait_us":1,"thread_start_us":769,"threads_started":1}
I20260929 15:52:10.721320 7671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:10.684377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33067 (local address 127.0.97.2:53229)
0929 15:52:10.699592 (+ 15215us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:10.699613 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:10.699741 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:10.700478 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:10.700487 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:10.701431 (+ 944us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:10.702526 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:10.702536 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:10.704255 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:10.704260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:10.707286 (+ 3026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:10.707296 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:10.709436 (+ 2140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:10.709442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:10.709619 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:10.710590 (+ 971us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:10.710618 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:10.713463 (+ 2845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:10.717626 (+ 4163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:10.717630 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:10.717634 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:10.718096 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:10.718533 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:10.718538 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:10.718540 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:10.718689 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:10.719319 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:10.719323 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:10.719468 (+ 145us) client_negotiation.cc:769] Sending connection context
0929 15:52:10.719609 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 15:52:10.719836 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15066,"mutex_wait_us":20,"thread_start_us":14104,"threads_started":1}
I20260929 15:52:10.722450 7673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:10.684474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53229 (local address 127.0.97.62:33067)
0929 15:52:10.698418 (+ 13944us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:10.698427 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:10.699717 (+ 1290us) server_negotiation.cc:407] Connection header received
0929 15:52:10.699903 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:10.699911 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:10.700029 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:10.700222 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:10.702695 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:10.704091 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:10.708371 (+ 4280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:10.709063 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:10.709191 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:10.713738 (+ 4547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:10.713745 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:10.713750 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:10.713799 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:10.717479 (+ 3680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:10.718237 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:10.718243 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:10.718246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:10.718395 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:10.718822 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:10.718828 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:10.718831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:10.719133 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:10.719262 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:10.721851 (+ 2589us) server_negotiation.cc:299] Negotiation successful
0929 15:52:10.722064 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13791,"thread_start_us":13687,"threads_started":1}
I20260929 15:52:10.722564 7669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:10.681570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43621 (local address 127.0.97.60:36403)
0929 15:52:10.701158 (+ 19588us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:10.701165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:10.701188 (+ 23us) server_negotiation.cc:407] Connection header received
0929 15:52:10.701242 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:10.701248 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:10.701330 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:10.703758 (+ 2428us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:10.703965 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:10.705139 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:10.707131 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:10.707782 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:10.708653 (+ 871us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:10.712219 (+ 3566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:10.712225 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:10.712229 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:10.712256 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:10.716111 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:10.717043 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:10.717048 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:10.717051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:10.717191 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:10.717783 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:10.717788 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:10.717791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:10.718136 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:10.721625 (+ 3489us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:10.721752 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 15:52:10.721982 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19451,"thread_start_us":18420,"threads_started":1}
I20260929 15:52:10.723992 7664 heartbeater.cc:347] Connected to a master server at 127.0.97.60:36403
I20260929 15:52:10.724134 7665 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33067
I20260929 15:52:10.724323 7665 heartbeater.cc:464] Registering TS with master...
I20260929 15:52:10.724319 7664 heartbeater.cc:464] Registering TS with master...
I20260929 15:52:10.725373 7665 heartbeater.cc:511] Master 127.0.97.62:33067 requested a full tablet report, sending...
I20260929 15:52:10.725400 7664 heartbeater.cc:511] Master 127.0.97.60:36403 requested a full tablet report, sending...
I20260929 15:52:10.726974 6551 ts_manager.cc:194] Re-registered known tserver with Master: 2f0aa7392be545f89274d4a92743c022 (127.0.97.2:42699)
I20260929 15:52:10.727072 7668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:10.681573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41675 (local address 127.0.97.61:41069)
0929 15:52:10.700574 (+ 19001us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:10.700579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:10.700594 (+ 15us) server_negotiation.cc:407] Connection header received
0929 15:52:10.700638 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:10.700641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:10.700692 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:10.700799 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:10.702230 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:10.703398 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:10.707432 (+ 4034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:10.708050 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:10.708168 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:10.716101 (+ 7933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:10.716109 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:10.716115 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:10.716153 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:10.723454 (+ 7301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:10.724969 (+ 1515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:10.724976 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:10.724979 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:10.725114 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:10.725569 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:10.725577 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:10.725583 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:10.725926 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:10.726078 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:10.726537 (+ 459us) server_negotiation.cc:299] Negotiation successful
0929 15:52:10.726743 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18904,"thread_start_us":18788,"threads_started":1}
I20260929 15:52:10.727094 7672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:10.682308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41069 (local address 127.0.97.2:41675)
0929 15:52:10.698824 (+ 16516us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:10.698854 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:10.699004 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:10.700847 (+ 1843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:10.700854 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:10.701435 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:10.702056 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:10.702073 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:10.704006 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:10.704011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:10.706960 (+ 2949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:10.706969 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:10.710000 (+ 3031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:10.710007 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:10.710194 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:10.710757 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:10.710779 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:10.715884 (+ 5105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:10.724334 (+ 8450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:10.724341 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:10.724345 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:10.724829 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:10.725254 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:10.725258 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:10.725262 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:10.725438 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:10.726134 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:10.726138 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:10.726260 (+ 122us) client_negotiation.cc:769] Sending connection context
0929 15:52:10.726434 (+ 174us) client_negotiation.cc:240] Negotiation successful
0929 15:52:10.726692 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16305,"mutex_wait_us":1934,"thread_start_us":16229,"threads_started":1}
I20260929 15:52:10.727263 7675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:10.687447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41396 (local address 127.0.97.3:35181)
0929 15:52:10.691019 (+ 3572us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:10.691031 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:10.691064 (+ 33us) server_negotiation.cc:407] Connection header received
0929 15:52:10.691145 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:10.691156 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:10.692307 (+ 1151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:10.695205 (+ 2898us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:10.695524 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:10.697449 (+ 1925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:10.706132 (+ 8683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:10.706949 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:10.707082 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:10.716936 (+ 9854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:10.716952 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:10.716971 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:10.717013 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:10.720505 (+ 3492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:10.724369 (+ 3864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:10.724379 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:10.724385 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:10.724600 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:10.725086 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:10.725092 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:10.725095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:10.725533 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:10.725803 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:10.726578 (+ 775us) server_negotiation.cc:299] Negotiation successful
0929 15:52:10.726818 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3375,"thread_start_us":3202,"threads_started":1}
I20260929 15:52:10.728096 6551 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53229
I20260929 15:52:10.728605 7666 heartbeater.cc:347] Connected to a master server at 127.0.97.61:41069
I20260929 15:52:10.728755 7666 heartbeater.cc:464] Registering TS with master...
I20260929 15:52:10.728806 7339 ts_manager.cc:194] Registered new tserver with Master: 2f0aa7392be545f89274d4a92743c022 (127.0.97.2:42699)
I20260929 15:52:10.729256 7666 heartbeater.cc:511] Master 127.0.97.61:41069 requested a full tablet report, sending...
I20260929 15:52:10.729275 7002 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:52:10.730497 7148 ts_manager.cc:194] Registered new tserver with Master: 2f0aa7392be545f89274d4a92743c022 (127.0.97.2:42699)
I20260929 15:52:10.811048 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 6879
I20260929 15:52:10.827636 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:35181
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:33067
--webserver_port=40793
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:33067,127.0.97.61:41069
--tserver_master_addrs=127.0.97.62:33067,127.0.97.61:41069,127.0.97.60:36403 with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:52:11.145927 7518 heartbeater.cc:503] Master 127.0.97.62:33067 was elected leader, sending a full tablet report...
W20260929 15:52:11.156633 7677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:52:11.156960 7677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:52:11.157091 7677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:52:11.164427 7677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:52:11.164533 7677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:52:11.164642 7677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:52:11.174404 7677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:35181
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=40793
--tserver_master_addrs=127.0.97.62:33067,127.0.97.61:41069,127.0.97.60:36403
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:52:11.175582 7677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:52:11.176746 7677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:52:11.189673 7683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:11.189692 7684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:11.190353 7686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:52:11.194988 7677 server_base.cc:1034] running on GCE node
I20260929 15:52:11.195669 7677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:52:11.196683 7677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:52:11.197933 7677 hybrid_clock.cc:648] HybridClock initialized: now 1790697131197887 us; error 46 us; skew 500 ppm
I20260929 15:52:11.198294 7677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:52:11 dist-test-slave-r0ck krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697131, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:52:11.201555 7677 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260929 15:52:11.203105 7677 webserver.cc:460] Webserver started at http://127.0.97.3:40793/ using document root <none> and password file <none>
I20260929 15:52:11.203815 7677 fs_manager.cc:359] Metadata directory not provided
I20260929 15:52:11.203970 7677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:52:11.209383 7677 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:52:11.212348 7693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:11.213852 7677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:52:11.214046 7677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-2/wal
uuid: "6b634f743a52484a9f8f7978e146cbba"
format_stamp: "Formatted at 2026-09-29 15:52:08 on dist-test-slave-r0ck"
server_key: "a890323a08047d200c12c74e2f3eb704"
server_key_iv: "c7fdcf9e87ba04f082bf5491f056e6ab"
server_key_version: "encryptionkey@0"
I20260929 15:52:11.215049 7677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:52:11.232198 7677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:52:11.236680 7677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:52:11.237004 7677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:52:11.238138 7677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:52:11.239502 7677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:52:11.239624 7677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:11.239753 7677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:52:11.239836 7677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:11.274505 7677 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35181
I20260929 15:52:11.274592 7805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35181 every 8 connection(s)
I20260929 15:52:11.275966 7677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:52:11.279042 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 7677
I20260929 15:52:11.324306 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:11.288812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59436 (local address 127.0.97.61:41069)
0929 15:52:11.289374 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:11.289383 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:11.294868 (+ 5485us) server_negotiation.cc:407] Connection header received
0929 15:52:11.295090 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:11.295100 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:11.295207 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:11.295331 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:11.297371 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.298857 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.301535 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.302234 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.302372 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:11.306286 (+ 3914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:11.306292 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:11.306297 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:11.306352 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:11.312719 (+ 6367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:11.315141 (+ 2422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:11.315147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:11.315151 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:11.315355 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:11.316221 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:11.316228 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:11.316233 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:11.316611 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:11.316860 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:11.318084 (+ 1224us) server_negotiation.cc:299] Negotiation successful
0929 15:52:11.318326 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":301,"threads_started":1}
I20260929 15:52:11.329697 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:11.288371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40320 (local address 127.0.97.62:33067)
0929 15:52:11.288868 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:11.288876 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:11.299274 (+ 10398us) server_negotiation.cc:407] Connection header received
0929 15:52:11.299439 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:11.299444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:11.299545 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:11.299657 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:11.307022 (+ 7365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.308550 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.313153 (+ 4603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.314113 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.314268 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:11.318592 (+ 4324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:11.318603 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:11.318611 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:11.318668 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:11.323816 (+ 5148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:11.326759 (+ 2943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:11.326775 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:11.326778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:11.327042 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:11.327499 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:11.327507 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:11.327511 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:11.327871 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:11.328113 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:11.329034 (+ 921us) server_negotiation.cc:299] Negotiation successful
0929 15:52:11.329248 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":252,"threads_started":1}
Sep 29 15:52:11 dist-test-slave-r0ck krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697131, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 29 15:52:11 dist-test-slave-r0ck krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697131, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 15:52:11.334360 6551 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:52:11.344396 6551 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:52:11.346433 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:11.307461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58056 (local address 127.0.97.60:36403)
0929 15:52:11.308953 (+ 1492us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:11.308961 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:11.308984 (+ 23us) server_negotiation.cc:407] Connection header received
0929 15:52:11.309054 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:11.309062 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:11.309159 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:11.311749 (+ 2590us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:11.311936 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.313588 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.322792 (+ 9204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.323691 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.323860 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:11.338153 (+ 14293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:11.338164 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:11.338170 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:11.338214 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:11.342579 (+ 4365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:11.343405 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:11.343411 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:11.343414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:11.343581 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:11.344016 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:11.344021 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:11.344024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:11.344374 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:11.344606 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:11.345817 (+ 1211us) server_negotiation.cc:299] Negotiation successful
0929 15:52:11.346036 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1330,"thread_start_us":1818,"threads_started":1}
I20260929 15:52:11.347467 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:11.281534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43789 (local address 127.0.97.61:41069)
0929 15:52:11.285765 (+ 4231us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:11.285775 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:11.302615 (+ 16840us) server_negotiation.cc:407] Connection header received
0929 15:52:11.324934 (+ 22319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:11.324943 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:11.325056 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:11.325173 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:11.327946 (+ 2773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.329127 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.332720 (+ 3593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.333474 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.333580 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:11.339133 (+ 5553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:11.339139 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:11.339142 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:11.339180 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:11.343421 (+ 4241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:11.344381 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:11.344386 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:11.344390 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:11.344600 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:11.345146 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:11.345151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:11.345154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:11.345538 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:11.345661 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:11.346772 (+ 1111us) server_negotiation.cc:299] Negotiation successful
0929 15:52:11.347018 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4061,"thread_start_us":3952,"threads_started":1}
Sep 29 15:52:11 dist-test-slave-r0ck krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697131, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260929 15:52:11.350206 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:11.304010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33067 (local address 127.0.97.3:56429)
0929 15:52:11.328274 (+ 24264us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:11.328291 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:11.328434 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:11.330057 (+ 1623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:11.330060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:11.330711 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:11.331104 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:11.331121 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.332778 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.332783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:11.334767 (+ 1984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:11.334778 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.335657 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.335664 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:11.335859 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:11.337446 (+ 1587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:11.337475 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:11.341399 (+ 3924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:11.346055 (+ 4656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:11.346060 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:11.346063 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:11.346712 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:11.347383 (+ 671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:11.347388 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:11.347391 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:11.347801 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:11.348445 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:11.348452 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:11.348635 (+ 183us) client_negotiation.cc:769] Sending connection context
0929 15:52:11.348888 (+ 253us) client_negotiation.cc:240] Negotiation successful
0929 15:52:11.349176 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24159,"thread_start_us":4354,"threads_started":1}
I20260929 15:52:11.350257 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:11.279526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41069 (local address 127.0.97.3:43789)
0929 15:52:11.302390 (+ 22864us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:11.302459 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:11.315434 (+ 12975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:11.325996 (+ 10562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:11.326003 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:11.326652 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:11.327783 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:11.327798 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.329292 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.329299 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:11.332537 (+ 3238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:11.332552 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.334080 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.334087 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:11.334382 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:11.335922 (+ 1540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:11.335962 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:11.338868 (+ 2906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:11.343592 (+ 4724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:11.343602 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:11.343620 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:11.344234 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:11.344765 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:11.344770 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:11.344774 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:11.345013 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:11.345775 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:11.345784 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:11.346229 (+ 445us) client_negotiation.cc:769] Sending connection context
0929 15:52:11.347771 (+ 1542us) client_negotiation.cc:240] Negotiation successful
0929 15:52:11.348354 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22423,"thread_start_us":48382,"threads_started":1}
I20260929 15:52:11.351540 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:11.304488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56429 (local address 127.0.97.62:33067)
0929 15:52:11.327048 (+ 22560us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:11.327053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:11.328371 (+ 1318us) server_negotiation.cc:407] Connection header received
0929 15:52:11.329672 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:11.329677 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:11.329789 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:11.329949 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:11.331396 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.332611 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.334933 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.335459 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.335613 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:11.341677 (+ 6064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:11.341686 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:11.341691 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:11.341728 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:11.345912 (+ 4184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:11.346841 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:11.346846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:11.346849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:11.347248 (+ 399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:11.347931 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:11.347935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:11.347939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:11.348302 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:11.348446 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:11.349817 (+ 1371us) server_negotiation.cc:299] Negotiation successful
0929 15:52:11.350069 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22400,"thread_start_us":22574,"threads_started":1}
I20260929 15:52:11.352792 7807 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33067
I20260929 15:52:11.352779 7808 heartbeater.cc:347] Connected to a master server at 127.0.97.61:41069
I20260929 15:52:11.353440 7807 heartbeater.cc:464] Registering TS with master...
I20260929 15:52:11.353440 7808 heartbeater.cc:464] Registering TS with master...
I20260929 15:52:11.354424 7808 heartbeater.cc:511] Master 127.0.97.61:41069 requested a full tablet report, sending...
I20260929 15:52:11.355314 7807 heartbeater.cc:511] Master 127.0.97.62:33067 requested a full tablet report, sending...
I20260929 15:52:11.356038 7148 ts_manager.cc:194] Registered new tserver with Master: 6b634f743a52484a9f8f7978e146cbba (127.0.97.3:35181)
I20260929 15:52:11.356864 6550 ts_manager.cc:194] Re-registered known tserver with Master: 6b634f743a52484a9f8f7978e146cbba (127.0.97.3:35181)
I20260929 15:52:11.357976 6550 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:56429
I20260929 15:52:11.359362 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:11.279526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36403 (local address 127.0.97.3:49509)
0929 15:52:11.308144 (+ 28618us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:11.308169 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:11.315823 (+ 7654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:11.317043 (+ 1220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:11.317062 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:11.318306 (+ 1244us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:11.319496 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:11.319528 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.330973 (+ 11445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.330979 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:11.332626 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:11.332638 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.346452 (+ 13814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.346460 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:11.346642 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:11.347220 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:11.347241 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:11.350349 (+ 3108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:11.355799 (+ 5450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:11.355807 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:11.355814 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:11.356505 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:11.357091 (+ 586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:11.357100 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:11.357106 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:11.357283 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:11.358229 (+ 946us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:11.358238 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:11.358441 (+ 203us) client_negotiation.cc:769] Sending connection context
0929 15:52:11.358586 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 15:52:11.358907 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28419,"mutex_wait_us":411,"thread_start_us":22657,"threads_started":1}
I20260929 15:52:11.359678 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:11.279826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49509 (local address 127.0.97.60:36403)
0929 15:52:11.304587 (+ 24761us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:11.304597 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:11.308350 (+ 3753us) server_negotiation.cc:407] Connection header received
0929 15:52:11.316135 (+ 7785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:11.316144 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:11.316267 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:11.316382 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:11.327611 (+ 11229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.330657 (+ 3046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.345412 (+ 14755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.346090 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.346208 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:11.350611 (+ 4403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:11.350618 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:11.350622 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:11.350662 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:11.355619 (+ 4957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:11.356682 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:11.356691 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:11.356697 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:11.356879 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:11.357440 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:11.357447 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:11.357452 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:11.357906 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:11.358092 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:11.358943 (+ 851us) server_negotiation.cc:299] Negotiation successful
0929 15:52:11.359165 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24590,"thread_start_us":27402,"threads_started":1}
I20260929 15:52:11.360443 7806 heartbeater.cc:347] Connected to a master server at 127.0.97.60:36403
I20260929 15:52:11.360591 7806 heartbeater.cc:464] Registering TS with master...
I20260929 15:52:11.360996 7806 heartbeater.cc:511] Master 127.0.97.60:36403 requested a full tablet report, sending...
I20260929 15:52:11.364607 7339 ts_manager.cc:194] Registered new tserver with Master: 6b634f743a52484a9f8f7978e146cbba (127.0.97.3:35181)
I20260929 15:52:11.386634 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:11.378082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41412 (local address 127.0.97.3:35181)
0929 15:52:11.378734 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:11.378745 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:11.379026 (+ 281us) server_negotiation.cc:407] Connection header received
0929 15:52:11.379227 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:11.379241 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:11.379437 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:11.379664 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:52:11.380250 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.381800 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.383950 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.385614 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.385761 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:11.385969 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:11.386185 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 15:52:11.386338 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":481,"threads_started":1}
I20260929 15:52:11.386615 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:11.378442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35181 (local address 127.0.0.1:41412)
0929 15:52:11.378921 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:11.378934 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:11.379068 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:11.379659 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:11.379663 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:11.379686 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:52:11.380075 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:11.380085 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.381965 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.381969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:11.383399 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:11.383410 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.385809 (+ 2399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.385816 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:11.385980 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:11.386018 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 15:52:11.386136 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 15:52:11.386265 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":324,"threads_started":1}
I20260929 15:52:11.390254 7741 tablet_service.cc:1467] Processing CreateTablet for tablet be3347d3ed4a48f491d568918e96ede3 (DEFAULT_TABLE table=test-table [id=c93fc80e369d49dfa22daab9d5d89231]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:52:11.391484 7741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be3347d3ed4a48f491d568918e96ede3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:52:11.399729 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:11.377417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48198 (local address 127.0.97.2:42699)
0929 15:52:11.389052 (+ 11635us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:11.389064 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:11.389102 (+ 38us) server_negotiation.cc:407] Connection header received
0929 15:52:11.389226 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:11.389237 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:11.389706 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:11.391076 (+ 1370us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:52:11.391338 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.394247 (+ 2909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.397181 (+ 2934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.398584 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.398712 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:11.398929 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:11.399258 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 15:52:11.399421 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11411,"thread_start_us":12486,"threads_started":1}
I20260929 15:52:11.399744 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:11.377132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42699 (local address 127.0.0.1:48198)
0929 15:52:11.377727 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:11.377749 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:11.377870 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:11.390247 (+ 12377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:11.390256 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:11.390297 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:52:11.390813 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:11.390828 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.394412 (+ 3584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.394419 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:11.396553 (+ 2134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:11.396571 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.398827 (+ 2256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.398839 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:11.399061 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:11.399093 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 15:52:11.399197 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 15:52:11.399345 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":31,"thread_start_us":325,"threads_started":1}
I20260929 15:52:11.401728 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:11.380122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43917 (local address 127.0.0.1:46354)
0929 15:52:11.380705 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:11.380724 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:11.380827 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:11.394106 (+ 13279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:11.394111 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:11.394137 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:52:11.394514 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:11.394529 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.396378 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.396385 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:11.398887 (+ 2502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:11.398902 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.400464 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.400470 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:11.401055 (+ 585us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:11.401094 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 15:52:11.401189 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 15:52:11.401315 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":3,"thread_start_us":485,"threads_started":1}
I20260929 15:52:11.404340 7840 tablet_bootstrap.cc:492] T be3347d3ed4a48f491d568918e96ede3 P 6b634f743a52484a9f8f7978e146cbba: Bootstrap starting.
I20260929 15:52:11.403998 7599 tablet_service.cc:1467] Processing CreateTablet for tablet be3347d3ed4a48f491d568918e96ede3 (DEFAULT_TABLE table=test-table [id=c93fc80e369d49dfa22daab9d5d89231]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:52:11.405128 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:11.379455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46354 (local address 127.0.97.1:43917)
0929 15:52:11.393299 (+ 13844us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:11.393305 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:11.393334 (+ 29us) server_negotiation.cc:407] Connection header received
0929 15:52:11.393473 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:11.393480 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:11.393691 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:11.393893 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:52:11.394730 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.396205 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.399056 (+ 2851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.400258 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.400414 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:11.400609 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:11.401250 (+ 641us) server_negotiation.cc:299] Negotiation successful
0929 15:52:11.401451 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13687,"thread_start_us":13567,"threads_started":1}
I20260929 15:52:11.405433 7599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be3347d3ed4a48f491d568918e96ede3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:52:11.408418 7840 tablet_bootstrap.cc:654] T be3347d3ed4a48f491d568918e96ede3 P 6b634f743a52484a9f8f7978e146cbba: Neither blocks nor log segments found. Creating new log.
I20260929 15:52:11.408769 7452 tablet_service.cc:1467] Processing CreateTablet for tablet be3347d3ed4a48f491d568918e96ede3 (DEFAULT_TABLE table=test-table [id=c93fc80e369d49dfa22daab9d5d89231]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:52:11.409710 7840 log.cc:824] T be3347d3ed4a48f491d568918e96ede3 P 6b634f743a52484a9f8f7978e146cbba: Log is configured to *not* fsync() on all Append() calls
I20260929 15:52:11.410163 7452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be3347d3ed4a48f491d568918e96ede3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:52:11.413043 7840 tablet_bootstrap.cc:492] T be3347d3ed4a48f491d568918e96ede3 P 6b634f743a52484a9f8f7978e146cbba: No bootstrap required, opened a new log
I20260929 15:52:11.413314 7840 ts_tablet_manager.cc:1397] T be3347d3ed4a48f491d568918e96ede3 P 6b634f743a52484a9f8f7978e146cbba: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260929 15:52:11.418773 7842 tablet_bootstrap.cc:492] T be3347d3ed4a48f491d568918e96ede3 P 2f0aa7392be545f89274d4a92743c022: Bootstrap starting.
I20260929 15:52:11.422144 7842 tablet_bootstrap.cc:654] T be3347d3ed4a48f491d568918e96ede3 P 2f0aa7392be545f89274d4a92743c022: Neither blocks nor log segments found. Creating new log.
I20260929 15:52:11.423636 7842 log.cc:824] T be3347d3ed4a48f491d568918e96ede3 P 2f0aa7392be545f89274d4a92743c022: Log is configured to *not* fsync() on all Append() calls
I20260929 15:52:11.424619 7840 raft_consensus.cc:348] T be3347d3ed4a48f491d568918e96ede3 P 6b634f743a52484a9f8f7978e146cbba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0aa7392be545f89274d4a92743c022" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42699 } } peers { permanent_uuid: "6b634f743a52484a9f8f7978e146cbba" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35181 } } peers { permanent_uuid: "3ab5024e8d1a4d259eefb3f421e119fc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43917 } }
I20260929 15:52:11.425053 7840 raft_consensus.cc:374] T be3347d3ed4a48f491d568918e96ede3 P 6b634f743a52484a9f8f7978e146cbba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:52:11.425163 7840 raft_consensus.cc:729] T be3347d3ed4a48f491d568918e96ede3 P 6b634f743a52484a9f8f7978e146cbba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b634f743a52484a9f8f7978e146cbba, State: Initialized, Role: FOLLOWER
I20260929 15:52:11.425909 7840 consensus_queue.cc:260] T be3347d3ed4a48f491d568918e96ede3 P 6b634f743a52484a9f8f7978e146cbba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0aa7392be545f89274d4a92743c022" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42699 } } peers { permanent_uuid: "6b634f743a52484a9f8f7978e146cbba" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35181 } } peers { permanent_uuid: "3ab5024e8d1a4d259eefb3f421e119fc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43917 } }
I20260929 15:52:11.426970 7842 tablet_bootstrap.cc:492] T be3347d3ed4a48f491d568918e96ede3 P 2f0aa7392be545f89274d4a92743c022: No bootstrap required, opened a new log
I20260929 15:52:11.427229 7842 ts_tablet_manager.cc:1397] T be3347d3ed4a48f491d568918e96ede3 P 2f0aa7392be545f89274d4a92743c022: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260929 15:52:11.428001 7840 ts_tablet_manager.cc:1428] T be3347d3ed4a48f491d568918e96ede3 P 6b634f743a52484a9f8f7978e146cbba: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260929 15:52:11.428336 7807 heartbeater.cc:503] Master 127.0.97.62:33067 was elected leader, sending a full tablet report...
I20260929 15:52:11.432834 7844 tablet_bootstrap.cc:492] T be3347d3ed4a48f491d568918e96ede3 P 3ab5024e8d1a4d259eefb3f421e119fc: Bootstrap starting.
I20260929 15:52:11.435789 7844 tablet_bootstrap.cc:654] T be3347d3ed4a48f491d568918e96ede3 P 3ab5024e8d1a4d259eefb3f421e119fc: Neither blocks nor log segments found. Creating new log.
I20260929 15:52:11.437146 7844 log.cc:824] T be3347d3ed4a48f491d568918e96ede3 P 3ab5024e8d1a4d259eefb3f421e119fc: Log is configured to *not* fsync() on all Append() calls
I20260929 15:52:11.439774 7844 tablet_bootstrap.cc:492] T be3347d3ed4a48f491d568918e96ede3 P 3ab5024e8d1a4d259eefb3f421e119fc: No bootstrap required, opened a new log
I20260929 15:52:11.440027 7844 ts_tablet_manager.cc:1397] T be3347d3ed4a48f491d568918e96ede3 P 3ab5024e8d1a4d259eefb3f421e119fc: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 15:52:11.440438 7842 raft_consensus.cc:348] T be3347d3ed4a48f491d568918e96ede3 P 2f0aa7392be545f89274d4a92743c022 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0aa7392be545f89274d4a92743c022" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42699 } } peers { permanent_uuid: "6b634f743a52484a9f8f7978e146cbba" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35181 } } peers { permanent_uuid: "3ab5024e8d1a4d259eefb3f421e119fc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43917 } }
I20260929 15:52:11.440891 7842 raft_consensus.cc:374] T be3347d3ed4a48f491d568918e96ede3 P 2f0aa7392be545f89274d4a92743c022 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:52:11.441004 7842 raft_consensus.cc:729] T be3347d3ed4a48f491d568918e96ede3 P 2f0aa7392be545f89274d4a92743c022 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f0aa7392be545f89274d4a92743c022, State: Initialized, Role: FOLLOWER
I20260929 15:52:11.441682 7842 consensus_queue.cc:260] T be3347d3ed4a48f491d568918e96ede3 P 2f0aa7392be545f89274d4a92743c022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0aa7392be545f89274d4a92743c022" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42699 } } peers { permanent_uuid: "6b634f743a52484a9f8f7978e146cbba" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35181 } } peers { permanent_uuid: "3ab5024e8d1a4d259eefb3f421e119fc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43917 } }
I20260929 15:52:11.446095 7665 heartbeater.cc:503] Master 127.0.97.62:33067 was elected leader, sending a full tablet report...
I20260929 15:52:11.447062 7842 ts_tablet_manager.cc:1428] T be3347d3ed4a48f491d568918e96ede3 P 2f0aa7392be545f89274d4a92743c022: Time spent starting tablet: real 0.020s user 0.006s sys 0.009s
I20260929 15:52:11.448406 7844 raft_consensus.cc:348] T be3347d3ed4a48f491d568918e96ede3 P 3ab5024e8d1a4d259eefb3f421e119fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0aa7392be545f89274d4a92743c022" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42699 } } peers { permanent_uuid: "6b634f743a52484a9f8f7978e146cbba" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35181 } } peers { permanent_uuid: "3ab5024e8d1a4d259eefb3f421e119fc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43917 } }
I20260929 15:52:11.448838 7844 raft_consensus.cc:374] T be3347d3ed4a48f491d568918e96ede3 P 3ab5024e8d1a4d259eefb3f421e119fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:52:11.448946 7844 raft_consensus.cc:729] T be3347d3ed4a48f491d568918e96ede3 P 3ab5024e8d1a4d259eefb3f421e119fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ab5024e8d1a4d259eefb3f421e119fc, State: Initialized, Role: FOLLOWER
I20260929 15:52:11.449633 7844 consensus_queue.cc:260] T be3347d3ed4a48f491d568918e96ede3 P 3ab5024e8d1a4d259eefb3f421e119fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0aa7392be545f89274d4a92743c022" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42699 } } peers { permanent_uuid: "6b634f743a52484a9f8f7978e146cbba" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35181 } } peers { permanent_uuid: "3ab5024e8d1a4d259eefb3f421e119fc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43917 } }
I20260929 15:52:11.451453 7844 ts_tablet_manager.cc:1428] T be3347d3ed4a48f491d568918e96ede3 P 3ab5024e8d1a4d259eefb3f421e119fc: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
W20260929 15:52:11.535267 7809 tablet.cc:2369] T be3347d3ed4a48f491d568918e96ede3 P 6b634f743a52484a9f8f7978e146cbba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:52:11.555744 7848 raft_consensus.cc:482] T be3347d3ed4a48f491d568918e96ede3 P 3ab5024e8d1a4d259eefb3f421e119fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:52:11.556043 7848 raft_consensus.cc:504] T be3347d3ed4a48f491d568918e96ede3 P 3ab5024e8d1a4d259eefb3f421e119fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0aa7392be545f89274d4a92743c022" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42699 } } peers { permanent_uuid: "6b634f743a52484a9f8f7978e146cbba" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35181 } } peers { permanent_uuid: "3ab5024e8d1a4d259eefb3f421e119fc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43917 } }
I20260929 15:52:11.557780 7848 leader_election.cc:288] T be3347d3ed4a48f491d568918e96ede3 P 3ab5024e8d1a4d259eefb3f421e119fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f0aa7392be545f89274d4a92743c022 (127.0.97.2:42699), 6b634f743a52484a9f8f7978e146cbba (127.0.97.3:35181)
I20260929 15:52:11.565158 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:11.558041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42699 (local address 127.0.97.1:39121)
0929 15:52:11.558713 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:11.558734 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:11.558921 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:11.559316 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:11.559321 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:11.559355 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:52:11.559796 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:11.559810 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.561079 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.561084 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:11.563081 (+ 1997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:11.563093 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.564385 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.564391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:11.564572 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:11.564608 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 15:52:11.564713 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 15:52:11.564852 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":334,"threads_started":1}
I20260929 15:52:11.565152 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:11.558163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39121 (local address 127.0.97.2:42699)
0929 15:52:11.558423 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:11.558431 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:11.558833 (+ 402us) server_negotiation.cc:407] Connection header received
0929 15:52:11.559069 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:11.559076 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:11.559171 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:11.559275 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:52:11.560000 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.560947 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.563235 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.564193 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.564297 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:11.564364 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:11.564746 (+ 382us) server_negotiation.cc:299] Negotiation successful
0929 15:52:11.564906 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 15:52:11.565635 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:11.558776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35181 (local address 127.0.97.1:50903)
0929 15:52:11.559465 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:11.559482 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:11.559582 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:11.559988 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:11.559993 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:11.560018 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:52:11.560344 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:11.560355 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.561500 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.561503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:11.563576 (+ 2073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:11.563589 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.564940 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.564946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:11.565107 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:11.565140 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 15:52:11.565238 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 15:52:11.565384 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":445,"threads_started":1}
I20260929 15:52:11.565832 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:11.559034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50903 (local address 127.0.97.3:35181)
0929 15:52:11.559198 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:11.559203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:11.559572 (+ 369us) server_negotiation.cc:407] Connection header received
0929 15:52:11.559698 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:11.559706 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:11.559796 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:11.559913 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:52:11.560516 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.561333 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.563726 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.564716 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.564825 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:11.564914 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:11.565440 (+ 526us) server_negotiation.cc:299] Negotiation successful
0929 15:52:11.565573 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 15:52:11.566430 7619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be3347d3ed4a48f491d568918e96ede3" candidate_uuid: "3ab5024e8d1a4d259eefb3f421e119fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f0aa7392be545f89274d4a92743c022" is_pre_election: true
I20260929 15:52:11.566675 7761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be3347d3ed4a48f491d568918e96ede3" candidate_uuid: "3ab5024e8d1a4d259eefb3f421e119fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b634f743a52484a9f8f7978e146cbba" is_pre_election: true
I20260929 15:52:11.566902 7619 raft_consensus.cc:2454] T be3347d3ed4a48f491d568918e96ede3 P 2f0aa7392be545f89274d4a92743c022 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ab5024e8d1a4d259eefb3f421e119fc in term 0.
I20260929 15:52:11.567145 7761 raft_consensus.cc:2454] T be3347d3ed4a48f491d568918e96ede3 P 6b634f743a52484a9f8f7978e146cbba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ab5024e8d1a4d259eefb3f421e119fc in term 0.
I20260929 15:52:11.567585 7408 leader_election.cc:302] T be3347d3ed4a48f491d568918e96ede3 P 3ab5024e8d1a4d259eefb3f421e119fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f0aa7392be545f89274d4a92743c022, 3ab5024e8d1a4d259eefb3f421e119fc; no voters:
I20260929 15:52:11.567976 7848 raft_consensus.cc:2781] T be3347d3ed4a48f491d568918e96ede3 P 3ab5024e8d1a4d259eefb3f421e119fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:52:11.568140 7848 raft_consensus.cc:482] T be3347d3ed4a48f491d568918e96ede3 P 3ab5024e8d1a4d259eefb3f421e119fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:52:11.568279 7848 raft_consensus.cc:3037] T be3347d3ed4a48f491d568918e96ede3 P 3ab5024e8d1a4d259eefb3f421e119fc [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:52:11.570552 7848 raft_consensus.cc:504] T be3347d3ed4a48f491d568918e96ede3 P 3ab5024e8d1a4d259eefb3f421e119fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0aa7392be545f89274d4a92743c022" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42699 } } peers { permanent_uuid: "6b634f743a52484a9f8f7978e146cbba" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35181 } } peers { permanent_uuid: "3ab5024e8d1a4d259eefb3f421e119fc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43917 } }
I20260929 15:52:11.571092 7848 leader_election.cc:288] T be3347d3ed4a48f491d568918e96ede3 P 3ab5024e8d1a4d259eefb3f421e119fc [CANDIDATE]: Term 1 election: Requested vote from peers 2f0aa7392be545f89274d4a92743c022 (127.0.97.2:42699), 6b634f743a52484a9f8f7978e146cbba (127.0.97.3:35181)
I20260929 15:52:11.571524 7619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be3347d3ed4a48f491d568918e96ede3" candidate_uuid: "3ab5024e8d1a4d259eefb3f421e119fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f0aa7392be545f89274d4a92743c022"
I20260929 15:52:11.571589 7761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be3347d3ed4a48f491d568918e96ede3" candidate_uuid: "3ab5024e8d1a4d259eefb3f421e119fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b634f743a52484a9f8f7978e146cbba"
I20260929 15:52:11.571740 7619 raft_consensus.cc:3037] T be3347d3ed4a48f491d568918e96ede3 P 2f0aa7392be545f89274d4a92743c022 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:52:11.571781 7761 raft_consensus.cc:3037] T be3347d3ed4a48f491d568918e96ede3 P 6b634f743a52484a9f8f7978e146cbba [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:52:11.575417 7761 raft_consensus.cc:2454] T be3347d3ed4a48f491d568918e96ede3 P 6b634f743a52484a9f8f7978e146cbba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ab5024e8d1a4d259eefb3f421e119fc in term 1.
I20260929 15:52:11.575417 7619 raft_consensus.cc:2454] T be3347d3ed4a48f491d568918e96ede3 P 2f0aa7392be545f89274d4a92743c022 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ab5024e8d1a4d259eefb3f421e119fc in term 1.
I20260929 15:52:11.575774 7408 leader_election.cc:302] T be3347d3ed4a48f491d568918e96ede3 P 3ab5024e8d1a4d259eefb3f421e119fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f0aa7392be545f89274d4a92743c022, 3ab5024e8d1a4d259eefb3f421e119fc; no voters:
I20260929 15:52:11.576058 7848 raft_consensus.cc:2781] T be3347d3ed4a48f491d568918e96ede3 P 3ab5024e8d1a4d259eefb3f421e119fc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:52:11.576467 7848 raft_consensus.cc:686] T be3347d3ed4a48f491d568918e96ede3 P 3ab5024e8d1a4d259eefb3f421e119fc [term 1 LEADER]: Becoming Leader. State: Replica: 3ab5024e8d1a4d259eefb3f421e119fc, State: Running, Role: LEADER
I20260929 15:52:11.577005 7848 consensus_queue.cc:237] T be3347d3ed4a48f491d568918e96ede3 P 3ab5024e8d1a4d259eefb3f421e119fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0aa7392be545f89274d4a92743c022" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42699 } } peers { permanent_uuid: "6b634f743a52484a9f8f7978e146cbba" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35181 } } peers { permanent_uuid: "3ab5024e8d1a4d259eefb3f421e119fc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43917 } }
I20260929 15:52:11.581851 6551 catalog_manager.cc:5495] T be3347d3ed4a48f491d568918e96ede3 P 3ab5024e8d1a4d259eefb3f421e119fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ab5024e8d1a4d259eefb3f421e119fc (127.0.97.1). New cstate: current_term: 1 leader_uuid: "3ab5024e8d1a4d259eefb3f421e119fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0aa7392be545f89274d4a92743c022" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b634f743a52484a9f8f7978e146cbba" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ab5024e8d1a4d259eefb3f421e119fc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43917 } health_report { overall_health: HEALTHY } } }
W20260929 15:52:11.623165 7525 tablet.cc:2369] T be3347d3ed4a48f491d568918e96ede3 P 3ab5024e8d1a4d259eefb3f421e119fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:52:11.628017 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:11.622459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46364 (local address 127.0.97.1:43917)
0929 15:52:11.622668 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:11.622674 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:11.622694 (+ 20us) server_negotiation.cc:407] Connection header received
0929 15:52:11.622807 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:11.622810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:11.622883 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:11.623042 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:52:11.623464 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.624476 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.625956 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.626351 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.626449 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:11.626949 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:52:11.627420 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:52:11.627532 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:11.627630 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 15:52:11.627692 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 15:52:11.636405 7619 raft_consensus.cc:1261] T be3347d3ed4a48f491d568918e96ede3 P 2f0aa7392be545f89274d4a92743c022 [term 1 FOLLOWER]: Refusing update from remote peer 3ab5024e8d1a4d259eefb3f421e119fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:52:11.636404 7761 raft_consensus.cc:1261] T be3347d3ed4a48f491d568918e96ede3 P 6b634f743a52484a9f8f7978e146cbba [term 1 FOLLOWER]: Refusing update from remote peer 3ab5024e8d1a4d259eefb3f421e119fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:52:11.637387 7851 consensus_queue.cc:1035] T be3347d3ed4a48f491d568918e96ede3 P 3ab5024e8d1a4d259eefb3f421e119fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f0aa7392be545f89274d4a92743c022" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:52:11.637748 7848 consensus_queue.cc:1035] T be3347d3ed4a48f491d568918e96ede3 P 3ab5024e8d1a4d259eefb3f421e119fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b634f743a52484a9f8f7978e146cbba" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:52:11.650406 7855 mvcc.cc:205] Tried to move back new op lower bound from 7334695451172851712 to 7334695450943348736. Current Snapshot: MvccSnapshot[applied={T|T < 7334695451172851712}]
I20260929 15:52:11.663203 7856 mvcc.cc:205] Tried to move back new op lower bound from 7334695451172851712 to 7334695450943348736. Current Snapshot: MvccSnapshot[applied={T|T < 7334695451172851712}]
I20260929 15:52:11.676919 6550 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:52:11.677222 6550 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:52:11.685941 6550 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e21605ed4e8e4e9189d58164059f7de4: Sending DeleteTablet for 3 replicas of tablet be3347d3ed4a48f491d568918e96ede3
I20260929 15:52:11.687070 7741 tablet_service.cc:1514] Processing DeleteTablet for tablet be3347d3ed4a48f491d568918e96ede3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:52:11 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41412
I20260929 15:52:11.687156 7599 tablet_service.cc:1514] Processing DeleteTablet for tablet be3347d3ed4a48f491d568918e96ede3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:52:11 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48198
I20260929 15:52:11.689095 7452 tablet_service.cc:1514] Processing DeleteTablet for tablet be3347d3ed4a48f491d568918e96ede3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:52:11 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46354
I20260929 15:52:11.701076 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 6519
I20260929 15:52:11.702126 6581 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:52:11.707379 7864 tablet_replica.cc:330] T be3347d3ed4a48f491d568918e96ede3 P 6b634f743a52484a9f8f7978e146cbba: stopping tablet replica
I20260929 15:52:11.707836 7864 raft_consensus.cc:2229] T be3347d3ed4a48f491d568918e96ede3 P 6b634f743a52484a9f8f7978e146cbba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:52:11.708313 7864 raft_consensus.cc:2258] T be3347d3ed4a48f491d568918e96ede3 P 6b634f743a52484a9f8f7978e146cbba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:52:11.712510 7864 ts_tablet_manager.cc:1905] T be3347d3ed4a48f491d568918e96ede3 P 6b634f743a52484a9f8f7978e146cbba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:52:11.727512 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:11.689666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59442 (local address 127.0.97.61:41069)
0929 15:52:11.689848 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:11.689858 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:11.699520 (+ 9662us) server_negotiation.cc:407] Connection header received
0929 15:52:11.699730 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:11.699735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:11.699830 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:11.699931 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:11.703934 (+ 4003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.705075 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.709138 (+ 4063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.710016 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.710171 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:11.719657 (+ 9486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:11.719667 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:11.719675 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:11.719733 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:11.723500 (+ 3767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:11.724296 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:11.724301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:11.724304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:11.724513 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:11.724916 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:11.724921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:11.724925 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:11.725260 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:11.725506 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:11.726971 (+ 1465us) server_negotiation.cc:299] Negotiation successful
0929 15:52:11.727179 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 15:52:11.729801 7867 tablet_replica.cc:330] T be3347d3ed4a48f491d568918e96ede3 P 3ab5024e8d1a4d259eefb3f421e119fc: stopping tablet replica
I20260929 15:52:11.730294 7867 raft_consensus.cc:2229] T be3347d3ed4a48f491d568918e96ede3 P 3ab5024e8d1a4d259eefb3f421e119fc [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:52:11.730938 7867 raft_consensus.cc:2258] T be3347d3ed4a48f491d568918e96ede3 P 3ab5024e8d1a4d259eefb3f421e119fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:52:11.731669 7865 tablet_replica.cc:330] T be3347d3ed4a48f491d568918e96ede3 P 2f0aa7392be545f89274d4a92743c022: stopping tablet replica
I20260929 15:52:11.731953 7864 ts_tablet_manager.cc:1918] T be3347d3ed4a48f491d568918e96ede3 P 6b634f743a52484a9f8f7978e146cbba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:52:11.732064 7865 raft_consensus.cc:2229] T be3347d3ed4a48f491d568918e96ede3 P 2f0aa7392be545f89274d4a92743c022 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:52:11.732133 7864 log.cc:1196] T be3347d3ed4a48f491d568918e96ede3 P 6b634f743a52484a9f8f7978e146cbba: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-2/wal/wals/be3347d3ed4a48f491d568918e96ede3
I20260929 15:52:11.732539 7865 raft_consensus.cc:2258] T be3347d3ed4a48f491d568918e96ede3 P 2f0aa7392be545f89274d4a92743c022 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:52:11.732595 7864 ts_tablet_manager.cc:1939] T be3347d3ed4a48f491d568918e96ede3 P 6b634f743a52484a9f8f7978e146cbba: Deleting consensus metadata
I20260929 15:52:11.734395 7865 ts_tablet_manager.cc:1905] T be3347d3ed4a48f491d568918e96ede3 P 2f0aa7392be545f89274d4a92743c022: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:52:11.741124 7867 ts_tablet_manager.cc:1905] T be3347d3ed4a48f491d568918e96ede3 P 3ab5024e8d1a4d259eefb3f421e119fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:52:11.741590 7865 ts_tablet_manager.cc:1918] T be3347d3ed4a48f491d568918e96ede3 P 2f0aa7392be545f89274d4a92743c022: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:52:11.741742 7865 log.cc:1196] T be3347d3ed4a48f491d568918e96ede3 P 2f0aa7392be545f89274d4a92743c022: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-1/wal/wals/be3347d3ed4a48f491d568918e96ede3
I20260929 15:52:11.742287 7865 ts_tablet_manager.cc:1939] T be3347d3ed4a48f491d568918e96ede3 P 2f0aa7392be545f89274d4a92743c022: Deleting consensus metadata
I20260929 15:52:11.745539 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:11.700180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58070 (local address 127.0.97.60:36403)
0929 15:52:11.700352 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:11.700358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:11.726688 (+ 26330us) server_negotiation.cc:407] Connection header received
0929 15:52:11.726876 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:11.726882 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:11.726991 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:11.727093 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:11.729984 (+ 2891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.731092 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.734327 (+ 3235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.734983 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.735088 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:11.738802 (+ 3714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:11.738809 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:11.738814 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:11.738855 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:11.742591 (+ 3736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:11.743365 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:11.743371 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:11.743374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:11.743549 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:11.744111 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:11.744116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:11.744120 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:11.744381 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:11.744634 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:11.744947 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 15:52:11.745163 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 15:52:11.751892 7867 ts_tablet_manager.cc:1918] T be3347d3ed4a48f491d568918e96ede3 P 3ab5024e8d1a4d259eefb3f421e119fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:52:11.752101 7867 log.cc:1196] T be3347d3ed4a48f491d568918e96ede3 P 3ab5024e8d1a4d259eefb3f421e119fc: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-0/wal/wals/be3347d3ed4a48f491d568918e96ede3
I20260929 15:52:11.752652 7867 ts_tablet_manager.cc:1939] T be3347d3ed4a48f491d568918e96ede3 P 3ab5024e8d1a4d259eefb3f421e119fc: Deleting consensus metadata
I20260929 15:52:11.870738 6538 catalog_manager.cc:4841] TS 3ab5024e8d1a4d259eefb3f421e119fc (127.0.97.1:43917): tablet be3347d3ed4a48f491d568918e96ede3 (table test-table [id=c93fc80e369d49dfa22daab9d5d89231]) successfully deleted
I20260929 15:52:11.871783 6538 catalog_manager.cc:4841] TS 2f0aa7392be545f89274d4a92743c022 (127.0.97.2:42699): tablet be3347d3ed4a48f491d568918e96ede3 (table test-table [id=c93fc80e369d49dfa22daab9d5d89231]) successfully deleted
I20260929 15:52:11.872138 6538 catalog_manager.cc:4841] TS 6b634f743a52484a9f8f7978e146cbba (127.0.97.3:35181): tablet be3347d3ed4a48f491d568918e96ede3 (table test-table [id=c93fc80e369d49dfa22daab9d5d89231]) successfully deleted
I20260929 15:52:11.872787 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 6519
I20260929 15:52:11.920323 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:11.903872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59456 (local address 127.0.97.61:41069)
0929 15:52:11.904130 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:11.904139 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:11.905092 (+ 953us) server_negotiation.cc:407] Connection header received
0929 15:52:11.905380 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:11.905388 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:11.905486 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:11.905575 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:11.907759 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.908866 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.910378 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.910803 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.910966 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:11.913956 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:11.913961 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:11.913964 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:11.913994 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:11.917399 (+ 3405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:11.918258 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:11.918265 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:11.918270 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:11.918411 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:11.918842 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:11.918849 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:11.918852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:11.919289 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:11.919526 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:11.919834 (+ 308us) server_negotiation.cc:299] Negotiation successful
0929 15:52:11.920038 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 15:52:11.922461 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:11.904849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58084 (local address 127.0.97.60:36403)
0929 15:52:11.905014 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:11.905020 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:11.905862 (+ 842us) server_negotiation.cc:407] Connection header received
0929 15:52:11.906007 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:11.906012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:11.906086 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:11.906192 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:11.908397 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.909236 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.911243 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:11.911781 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:11.911935 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:11.915078 (+ 3143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:11.915088 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:11.915093 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:11.915131 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:11.918444 (+ 3313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:11.919259 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:11.919267 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:11.919271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:11.919425 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:11.919846 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:11.919851 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:11.919855 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:11.920499 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:11.920786 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:11.921863 (+ 1077us) server_negotiation.cc:299] Negotiation successful
0929 15:52:11.922076 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 15:52:12.666314 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:12.665628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33067 (local address 127.0.97.1:33443)
0929 15:52:12.666034 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:12.666152 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":223,"threads_started":1}
W20260929 15:52:12.666742 7518 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:33067 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:33067: connect: Connection refused (error 111)
I20260929 15:52:12.668501 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:12.667703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33067 (local address 127.0.97.2:44207)
0929 15:52:12.668184 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:12.668309 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":236,"threads_started":1}
W20260929 15:52:12.668987 7665 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:33067 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:33067: connect: Connection refused (error 111)
I20260929 15:52:12.678610 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:12.677810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33067 (local address 127.0.97.3:42035)
0929 15:52:12.678235 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:12.678383 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":212,"threads_started":1}
W20260929 15:52:12.678969 7807 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:33067 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:33067: connect: Connection refused (error 111)
I20260929 15:52:13.184248 7878 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e21605ed4e8e4e9189d58164059f7de4)
I20260929 15:52:13.184504 7878 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: false } } peers { permanent_uuid: "df4d8c2416f947dca2364e0321d98599" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attrs { promote: false } }
I20260929 15:52:13.185925 7878 leader_election.cc:288] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e21605ed4e8e4e9189d58164059f7de4 (127.0.97.62:33067), df4d8c2416f947dca2364e0321d98599 (127.0.97.60:36403)
I20260929 15:52:13.186348 7879 raft_consensus.cc:482] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e21605ed4e8e4e9189d58164059f7de4)
I20260929 15:52:13.186589 7879 raft_consensus.cc:504] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: false } } peers { permanent_uuid: "df4d8c2416f947dca2364e0321d98599" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attrs { promote: false } }
I20260929 15:52:13.187266 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.186217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33067 (local address 127.0.0.1:40716)
0929 15:52:13.186727 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:13.186937 (+ 210us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":245,"threads_started":1}
W20260929 15:52:13.188550 7135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33067: connect: Connection refused (error 111)
I20260929 15:52:13.189293 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.188234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33067 (local address 127.0.0.1:40730)
0929 15:52:13.188803 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:13.189000 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":287,"threads_started":1}
W20260929 15:52:13.189556 7326 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33067: connect: Connection refused (error 111)
I20260929 15:52:13.198140 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.186838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36403 (local address 127.0.0.1:58100)
0929 15:52:13.187573 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:13.187596 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:13.187777 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:13.189062 (+ 1285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:13.189071 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:13.189118 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:52:13.189441 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:13.189452 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:13.191774 (+ 2322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:13.191783 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:13.194508 (+ 2725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:13.194528 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:13.197092 (+ 2564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:13.197101 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:13.197414 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:13.197461 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 15:52:13.197532 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 15:52:13.197690 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":1391,"threads_started":1}
I20260929 15:52:13.199487 7879 leader_election.cc:288] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e21605ed4e8e4e9189d58164059f7de4 (127.0.97.62:33067), b2ca5a81736e4a508dea4e247fce6a74 (127.0.97.61:41069)
I20260929 15:52:13.199924 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.187274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58100 (local address 127.0.97.60:36403)
0929 15:52:13.188327 (+ 1053us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:13.188333 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:13.188351 (+ 18us) server_negotiation.cc:407] Connection header received
0929 15:52:13.188414 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:13.188419 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:13.188489 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:13.189612 (+ 1123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:52:13.189898 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:13.191547 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:13.194775 (+ 3228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:13.196681 (+ 1906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:13.199249 (+ 2568us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:13.199348 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:13.199398 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 15:52:13.199543 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":940,"thread_start_us":757,"threads_started":1}
I20260929 15:52:13.201215 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.200581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33067 (local address 127.0.0.1:40744)
0929 15:52:13.200757 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:13.200928 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 15:52:13.201422 7348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2ca5a81736e4a508dea4e247fce6a74" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "df4d8c2416f947dca2364e0321d98599" is_pre_election: true
I20260929 15:52:13.201817 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.201444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33067 (local address 127.0.0.1:40756)
0929 15:52:13.201574 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:13.201654 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 15:52:13.201797 7348 raft_consensus.cc:2454] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2ca5a81736e4a508dea4e247fce6a74 in term 1.
W20260929 15:52:13.201668 7326 leader_election.cc:334] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e21605ed4e8e4e9189d58164059f7de4 (127.0.97.62:33067): Network error: Client connection negotiation failed: client connection to 127.0.97.62:33067: connect: Connection refused (error 111)
W20260929 15:52:13.202200 7135 leader_election.cc:334] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e21605ed4e8e4e9189d58164059f7de4 (127.0.97.62:33067): Network error: Client connection negotiation failed: client connection to 127.0.97.62:33067: connect: Connection refused (error 111)
I20260929 15:52:13.204172 7135 leader_election.cc:302] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b2ca5a81736e4a508dea4e247fce6a74, df4d8c2416f947dca2364e0321d98599; no voters: e21605ed4e8e4e9189d58164059f7de4
I20260929 15:52:13.204591 7878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 15:52:13.204720 7878 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e21605ed4e8e4e9189d58164059f7de4)
I20260929 15:52:13.204819 7878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [term 1 FOLLOWER]: Advancing to term 2
I20260929 15:52:13.206238 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.189086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41069 (local address 127.0.0.1:59472)
0929 15:52:13.195597 (+ 6511us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:13.195621 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:13.195786 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:13.197691 (+ 1905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:13.197695 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:13.197728 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:52:13.198083 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:13.198095 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:13.200902 (+ 2807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:13.200909 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:13.203862 (+ 2953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:13.203876 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:13.205667 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:13.205671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:13.205795 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:13.205819 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 15:52:13.205922 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 15:52:13.206016 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5968,"thread_start_us":11174,"threads_started":1}
I20260929 15:52:13.206429 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.189405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59472 (local address 127.0.97.61:41069)
0929 15:52:13.196736 (+ 7331us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:13.196742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:13.196764 (+ 22us) server_negotiation.cc:407] Connection header received
0929 15:52:13.196823 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:13.196829 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:13.197524 (+ 695us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:13.198422 (+ 898us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:52:13.198668 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:13.200120 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:13.204106 (+ 3986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:13.205511 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:13.205648 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:13.205760 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:13.205951 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 15:52:13.206079 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7219,"thread_start_us":7036,"threads_started":1}
I20260929 15:52:13.207082 7157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "df4d8c2416f947dca2364e0321d98599" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b2ca5a81736e4a508dea4e247fce6a74" is_pre_election: true
I20260929 15:52:13.207864 7878 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: false } } peers { permanent_uuid: "df4d8c2416f947dca2364e0321d98599" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attrs { promote: false } }
I20260929 15:52:13.208276 7157 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df4d8c2416f947dca2364e0321d98599 in current term 2: Already voted for candidate b2ca5a81736e4a508dea4e247fce6a74 in this term.
I20260929 15:52:13.208561 7878 leader_election.cc:288] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [CANDIDATE]: Term 2 election: Requested vote from peers e21605ed4e8e4e9189d58164059f7de4 (127.0.97.62:33067), df4d8c2416f947dca2364e0321d98599 (127.0.97.60:36403)
I20260929 15:52:13.208925 7323 leader_election.cc:302] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df4d8c2416f947dca2364e0321d98599; no voters: b2ca5a81736e4a508dea4e247fce6a74, e21605ed4e8e4e9189d58164059f7de4
I20260929 15:52:13.209132 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.208753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33067 (local address 127.0.0.1:40762)
0929 15:52:13.208896 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:13.209003 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 15:52:13.209275 7348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2ca5a81736e4a508dea4e247fce6a74" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "df4d8c2416f947dca2364e0321d98599"
I20260929 15:52:13.209574 7348 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599 [term 1 FOLLOWER]: Advancing to term 2
I20260929 15:52:13.210125 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.209746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33067 (local address 127.0.0.1:40770)
0929 15:52:13.209912 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:13.209990 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 15:52:13.210282 7135 leader_election.cc:334] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e21605ed4e8e4e9189d58164059f7de4 (127.0.97.62:33067): Network error: Client connection negotiation failed: client connection to 127.0.97.62:33067: connect: Connection refused (error 111)
I20260929 15:52:13.211819 7348 raft_consensus.cc:2454] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2ca5a81736e4a508dea4e247fce6a74 in term 2.
I20260929 15:52:13.211985 7879 raft_consensus.cc:2726] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 15:52:13.212329 7135 leader_election.cc:302] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b2ca5a81736e4a508dea4e247fce6a74, df4d8c2416f947dca2364e0321d98599; no voters: e21605ed4e8e4e9189d58164059f7de4
I20260929 15:52:13.212602 7878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 15:52:13.212975 7878 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [term 2 LEADER]: Becoming Leader. State: Replica: b2ca5a81736e4a508dea4e247fce6a74, State: Running, Role: LEADER
I20260929 15:52:13.213490 7878 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: false } } peers { permanent_uuid: "df4d8c2416f947dca2364e0321d98599" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attrs { promote: false } }
I20260929 15:52:13.215189 7888 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2ca5a81736e4a508dea4e247fce6a74. Latest consensus state: current_term: 2 leader_uuid: "b2ca5a81736e4a508dea4e247fce6a74" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: false } } peers { permanent_uuid: "df4d8c2416f947dca2364e0321d98599" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attrs { promote: false } } }
I20260929 15:52:13.215634 7888 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [sys.catalog]: This master's current role is: LEADER
I20260929 15:52:13.216717 7890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:52:13.220577 7890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:52:13.221441 7890 catalog_manager.cc:1259] Loaded cluster ID: 157dc853f93b4db599b14cf25dcddf85
I20260929 15:52:13.221532 7890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:52:13.222649 7890 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:52:13.223496 7890 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74: Loaded TSK: 0
I20260929 15:52:13.224318 7890 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:52:13.260215 7148 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:52:13.267119 7148 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:52:13.271549 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.271086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33067 (local address 127.0.0.1:40788)
0929 15:52:13.271282 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:13.271378 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 15:52:13.271584 7348 raft_consensus.cc:1261] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599 [term 2 FOLLOWER]: Refusing update from remote peer b2ca5a81736e4a508dea4e247fce6a74: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260929 15:52:13.272517 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.272133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33067 (local address 127.0.0.1:40802)
0929 15:52:13.272274 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:13.272341 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 15:52:13.272547 7888 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df4d8c2416f947dca2364e0321d98599" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260929 15:52:13.272812 7135 consensus_peers.cc:487] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 -> Peer e21605ed4e8e4e9189d58164059f7de4 (127.0.97.62:33067): Couldn't send request to peer e21605ed4e8e4e9189d58164059f7de4. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33067: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 15:52:13.275435 7879 sys_catalog.cc:455] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2ca5a81736e4a508dea4e247fce6a74. Latest consensus state: current_term: 2 leader_uuid: "b2ca5a81736e4a508dea4e247fce6a74" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: false } } peers { permanent_uuid: "df4d8c2416f947dca2364e0321d98599" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attrs { promote: false } } }
I20260929 15:52:13.275749 7879 sys_catalog.cc:458] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:52:13.277134 7878 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "b2ca5a81736e4a508dea4e247fce6a74" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: false } } peers { permanent_uuid: "df4d8c2416f947dca2364e0321d98599" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attrs { promote: false } } }
I20260929 15:52:13.277653 7878 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74 [sys.catalog]: This master's current role is: LEADER
I20260929 15:52:13.278802 7879 sys_catalog.cc:455] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "b2ca5a81736e4a508dea4e247fce6a74" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e21605ed4e8e4e9189d58164059f7de4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } peers { permanent_uuid: "b2ca5a81736e4a508dea4e247fce6a74" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41069 } attrs { promote: false } } peers { permanent_uuid: "df4d8c2416f947dca2364e0321d98599" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attrs { promote: false } } }
I20260929 15:52:13.279029 7879 sys_catalog.cc:458] T 00000000000000000000000000000000 P df4d8c2416f947dca2364e0321d98599 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:52:13.279420 7892 mvcc.cc:205] Tried to move back new op lower bound from 7334695457873227776 to 7334695457645957120. Current Snapshot: MvccSnapshot[applied={T|T < 7334695457873227776 or (T in {7334695457873227776})}]
I20260929 15:52:13.294369 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.286626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41414 (local address 127.0.97.3:35181)
0929 15:52:13.287196 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:13.287202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:13.288019 (+ 817us) server_negotiation.cc:407] Connection header received
0929 15:52:13.288308 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:13.288316 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:13.288433 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:13.288517 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:52:13.289617 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:13.290552 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:13.292436 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:13.293254 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:13.293334 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:13.293507 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:13.293953 (+ 446us) server_negotiation.cc:299] Negotiation successful
0929 15:52:13.294081 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":320,"threads_started":1}
I20260929 15:52:13.294405 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.287158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35181 (local address 127.0.0.1:41414)
0929 15:52:13.287908 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:13.287930 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:13.288129 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:13.288795 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:13.288802 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:13.288834 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:52:13.289397 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:13.289414 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:13.290737 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:13.290740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:13.292267 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:13.292279 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:13.293668 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:13.293672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:13.293793 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:13.293814 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 15:52:13.293930 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 15:52:13.294058 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":349,"threads_started":1}
I20260929 15:52:13.295344 7741 tablet_service.cc:1467] Processing CreateTablet for tablet 53842d40c4b949e7bc512667d213f345 (DEFAULT_TABLE table=test-table [id=015f2e180b5046f38193c20cb06b3113]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:52:13.295768 7741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53842d40c4b949e7bc512667d213f345. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:52:13.298081 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.286137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43917 (local address 127.0.0.1:46368)
0929 15:52:13.286321 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:13.286341 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:13.286459 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:13.290200 (+ 3741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:13.290205 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:13.290230 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:52:13.290519 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:13.290528 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:13.292304 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:13.292314 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:13.295088 (+ 2774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:13.295107 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:13.297293 (+ 2186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:13.297299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:13.297509 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:13.297544 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 15:52:13.297639 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 15:52:13.297781 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 15:52:13.298336 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.286219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46368 (local address 127.0.97.1:43917)
0929 15:52:13.289796 (+ 3577us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:13.289806 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:13.289832 (+ 26us) server_negotiation.cc:407] Connection header received
0929 15:52:13.289921 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:13.289929 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:13.290026 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:13.290160 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:52:13.290683 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:13.292090 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:13.295377 (+ 3287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:13.297027 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:13.297231 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:13.297415 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:13.297695 (+ 280us) server_negotiation.cc:299] Negotiation successful
0929 15:52:13.297871 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3424,"thread_start_us":3300,"threads_started":1}
I20260929 15:52:13.299194 7452 tablet_service.cc:1467] Processing CreateTablet for tablet 53842d40c4b949e7bc512667d213f345 (DEFAULT_TABLE table=test-table [id=015f2e180b5046f38193c20cb06b3113]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:52:13.299732 7452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53842d40c4b949e7bc512667d213f345. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:52:13.306324 7900 tablet_bootstrap.cc:492] T 53842d40c4b949e7bc512667d213f345 P 6b634f743a52484a9f8f7978e146cbba: Bootstrap starting.
I20260929 15:52:13.307156 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.285285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42699 (local address 127.0.0.1:48200)
0929 15:52:13.285447 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:13.285466 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:13.285572 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:13.301381 (+ 15809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:13.301386 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:13.301414 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:52:13.301586 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:13.301595 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:13.303096 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:13.303103 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:13.304897 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:13.304912 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:13.306475 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:13.306481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:13.306658 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:13.306690 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 15:52:13.306758 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 15:52:13.306901 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 15:52:13.307286 7901 tablet_bootstrap.cc:492] T 53842d40c4b949e7bc512667d213f345 P 3ab5024e8d1a4d259eefb3f421e119fc: Bootstrap starting.
I20260929 15:52:13.308073 7900 tablet_bootstrap.cc:654] T 53842d40c4b949e7bc512667d213f345 P 6b634f743a52484a9f8f7978e146cbba: Neither blocks nor log segments found. Creating new log.
I20260929 15:52:13.308190 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.285441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48200 (local address 127.0.97.2:42699)
0929 15:52:13.300935 (+ 15494us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:13.300942 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:13.300964 (+ 22us) server_negotiation.cc:407] Connection header received
0929 15:52:13.301030 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:13.301035 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:13.301111 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:13.301723 (+ 612us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:52:13.301869 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:13.302936 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:13.305099 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:13.306300 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:13.307745 (+ 1445us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:13.307832 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:13.307871 (+ 39us) server_negotiation.cc:299] Negotiation successful
0929 15:52:13.307966 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15329,"thread_start_us":15757,"threads_started":1}
I20260929 15:52:13.308840 7599 tablet_service.cc:1467] Processing CreateTablet for tablet 53842d40c4b949e7bc512667d213f345 (DEFAULT_TABLE table=test-table [id=015f2e180b5046f38193c20cb06b3113]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:52:13.309288 7599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53842d40c4b949e7bc512667d213f345. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:52:13.310237 7900 tablet_bootstrap.cc:492] T 53842d40c4b949e7bc512667d213f345 P 6b634f743a52484a9f8f7978e146cbba: No bootstrap required, opened a new log
I20260929 15:52:13.310379 7900 ts_tablet_manager.cc:1397] T 53842d40c4b949e7bc512667d213f345 P 6b634f743a52484a9f8f7978e146cbba: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 15:52:13.311120 7900 raft_consensus.cc:348] T 53842d40c4b949e7bc512667d213f345 P 6b634f743a52484a9f8f7978e146cbba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0aa7392be545f89274d4a92743c022" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42699 } } peers { permanent_uuid: "3ab5024e8d1a4d259eefb3f421e119fc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43917 } } peers { permanent_uuid: "6b634f743a52484a9f8f7978e146cbba" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35181 } }
I20260929 15:52:13.311347 7900 raft_consensus.cc:374] T 53842d40c4b949e7bc512667d213f345 P 6b634f743a52484a9f8f7978e146cbba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:52:13.311455 7900 raft_consensus.cc:729] T 53842d40c4b949e7bc512667d213f345 P 6b634f743a52484a9f8f7978e146cbba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b634f743a52484a9f8f7978e146cbba, State: Initialized, Role: FOLLOWER
I20260929 15:52:13.311686 7900 consensus_queue.cc:260] T 53842d40c4b949e7bc512667d213f345 P 6b634f743a52484a9f8f7978e146cbba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0aa7392be545f89274d4a92743c022" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42699 } } peers { permanent_uuid: "3ab5024e8d1a4d259eefb3f421e119fc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43917 } } peers { permanent_uuid: "6b634f743a52484a9f8f7978e146cbba" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35181 } }
I20260929 15:52:13.312412 7900 ts_tablet_manager.cc:1428] T 53842d40c4b949e7bc512667d213f345 P 6b634f743a52484a9f8f7978e146cbba: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 15:52:13.314910 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.312703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33067 (local address 127.0.97.3:43961)
0929 15:52:13.313260 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:13.313419 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":265,"threads_started":1}
I20260929 15:52:13.316291 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.315825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33067 (local address 127.0.97.3:50181)
0929 15:52:13.316039 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:13.316129 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260929 15:52:13.316712 7808 heartbeater.cc:503] Master 127.0.97.61:41069 was elected leader, sending a full tablet report...
I20260929 15:52:13.317513 7905 tablet_bootstrap.cc:492] T 53842d40c4b949e7bc512667d213f345 P 2f0aa7392be545f89274d4a92743c022: Bootstrap starting.
I20260929 15:52:13.317731 7901 tablet_bootstrap.cc:654] T 53842d40c4b949e7bc512667d213f345 P 3ab5024e8d1a4d259eefb3f421e119fc: Neither blocks nor log segments found. Creating new log.
I20260929 15:52:13.319295 7905 tablet_bootstrap.cc:654] T 53842d40c4b949e7bc512667d213f345 P 2f0aa7392be545f89274d4a92743c022: Neither blocks nor log segments found. Creating new log.
I20260929 15:52:13.320338 7901 tablet_bootstrap.cc:492] T 53842d40c4b949e7bc512667d213f345 P 3ab5024e8d1a4d259eefb3f421e119fc: No bootstrap required, opened a new log
I20260929 15:52:13.320465 7901 ts_tablet_manager.cc:1397] T 53842d40c4b949e7bc512667d213f345 P 3ab5024e8d1a4d259eefb3f421e119fc: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20260929 15:52:13.321321 7905 tablet_bootstrap.cc:492] T 53842d40c4b949e7bc512667d213f345 P 2f0aa7392be545f89274d4a92743c022: No bootstrap required, opened a new log
I20260929 15:52:13.321230 7901 raft_consensus.cc:348] T 53842d40c4b949e7bc512667d213f345 P 3ab5024e8d1a4d259eefb3f421e119fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0aa7392be545f89274d4a92743c022" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42699 } } peers { permanent_uuid: "3ab5024e8d1a4d259eefb3f421e119fc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43917 } } peers { permanent_uuid: "6b634f743a52484a9f8f7978e146cbba" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35181 } }
I20260929 15:52:13.321496 7905 ts_tablet_manager.cc:1397] T 53842d40c4b949e7bc512667d213f345 P 2f0aa7392be545f89274d4a92743c022: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20260929 15:52:13.321487 7901 raft_consensus.cc:374] T 53842d40c4b949e7bc512667d213f345 P 3ab5024e8d1a4d259eefb3f421e119fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:52:13.321588 7901 raft_consensus.cc:729] T 53842d40c4b949e7bc512667d213f345 P 3ab5024e8d1a4d259eefb3f421e119fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ab5024e8d1a4d259eefb3f421e119fc, State: Initialized, Role: FOLLOWER
I20260929 15:52:13.321848 7901 consensus_queue.cc:260] T 53842d40c4b949e7bc512667d213f345 P 3ab5024e8d1a4d259eefb3f421e119fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0aa7392be545f89274d4a92743c022" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42699 } } peers { permanent_uuid: "3ab5024e8d1a4d259eefb3f421e119fc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43917 } } peers { permanent_uuid: "6b634f743a52484a9f8f7978e146cbba" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35181 } }
I20260929 15:52:13.322280 7905 raft_consensus.cc:348] T 53842d40c4b949e7bc512667d213f345 P 2f0aa7392be545f89274d4a92743c022 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0aa7392be545f89274d4a92743c022" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42699 } } peers { permanent_uuid: "3ab5024e8d1a4d259eefb3f421e119fc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43917 } } peers { permanent_uuid: "6b634f743a52484a9f8f7978e146cbba" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35181 } }
I20260929 15:52:13.322593 7905 raft_consensus.cc:374] T 53842d40c4b949e7bc512667d213f345 P 2f0aa7392be545f89274d4a92743c022 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:52:13.322752 7905 raft_consensus.cc:729] T 53842d40c4b949e7bc512667d213f345 P 2f0aa7392be545f89274d4a92743c022 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f0aa7392be545f89274d4a92743c022, State: Initialized, Role: FOLLOWER
I20260929 15:52:13.323040 7905 consensus_queue.cc:260] T 53842d40c4b949e7bc512667d213f345 P 2f0aa7392be545f89274d4a92743c022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0aa7392be545f89274d4a92743c022" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42699 } } peers { permanent_uuid: "3ab5024e8d1a4d259eefb3f421e119fc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43917 } } peers { permanent_uuid: "6b634f743a52484a9f8f7978e146cbba" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35181 } }
I20260929 15:52:13.323487 7901 ts_tablet_manager.cc:1428] T 53842d40c4b949e7bc512667d213f345 P 3ab5024e8d1a4d259eefb3f421e119fc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 15:52:13.323884 7905 ts_tablet_manager.cc:1428] T 53842d40c4b949e7bc512667d213f345 P 2f0aa7392be545f89274d4a92743c022: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 15:52:13.327135 7666 heartbeater.cc:503] Master 127.0.97.61:41069 was elected leader, sending a full tablet report...
I20260929 15:52:13.332402 7519 heartbeater.cc:503] Master 127.0.97.61:41069 was elected leader, sending a full tablet report...
I20260929 15:52:13.334432 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.325337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33067 (local address 127.0.97.2:48871)
0929 15:52:13.333921 (+ 8584us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:13.334088 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8409,"thread_start_us":8236,"threads_started":1}
I20260929 15:52:13.334622 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.324565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33067 (local address 127.0.97.1:46639)
0929 15:52:13.334199 (+ 9634us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:13.334323 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9521,"threads_started":1}
I20260929 15:52:13.335647 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.335280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33067 (local address 127.0.97.1:47021)
0929 15:52:13.335459 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:13.335521 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 15:52:13.375353 7903 raft_consensus.cc:482] T 53842d40c4b949e7bc512667d213f345 P 6b634f743a52484a9f8f7978e146cbba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:52:13.375609 7903 raft_consensus.cc:504] T 53842d40c4b949e7bc512667d213f345 P 6b634f743a52484a9f8f7978e146cbba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0aa7392be545f89274d4a92743c022" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42699 } } peers { permanent_uuid: "3ab5024e8d1a4d259eefb3f421e119fc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43917 } } peers { permanent_uuid: "6b634f743a52484a9f8f7978e146cbba" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35181 } }
I20260929 15:52:13.377071 7903 leader_election.cc:288] T 53842d40c4b949e7bc512667d213f345 P 6b634f743a52484a9f8f7978e146cbba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f0aa7392be545f89274d4a92743c022 (127.0.97.2:42699), 3ab5024e8d1a4d259eefb3f421e119fc (127.0.97.1:43917)
I20260929 15:52:13.384131 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.377452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56441 (local address 127.0.97.2:42699)
0929 15:52:13.377670 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:13.377676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:13.377698 (+ 22us) server_negotiation.cc:407] Connection header received
0929 15:52:13.377773 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:13.377782 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:13.377879 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:13.378024 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:52:13.378539 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:13.379538 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:13.382108 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:13.383266 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:13.383400 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:13.383504 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:13.383793 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 15:52:13.383922 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 15:52:13.384294 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.377221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42699 (local address 127.0.97.3:56441)
0929 15:52:13.377446 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:13.377468 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:13.377631 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:13.378082 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:13.378088 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:13.378122 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:52:13.378403 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:13.378417 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:13.379673 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:13.379676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:13.381945 (+ 2269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:13.381959 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:13.383452 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:13.383456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:13.383601 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:13.383636 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 15:52:13.383816 (+ 180us) client_negotiation.cc:240] Negotiation successful
0929 15:52:13.383974 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":77}
I20260929 15:52:13.384543 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.377731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60247 (local address 127.0.97.1:43917)
0929 15:52:13.377987 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:13.377996 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:13.378338 (+ 342us) server_negotiation.cc:407] Connection header received
0929 15:52:13.378534 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:13.378539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:13.378628 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:13.378724 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:52:13.379113 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:13.379954 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:13.382327 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:13.383612 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:13.383703 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:13.383820 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:13.384138 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 15:52:13.384285 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260929 15:52:13.384742 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.377629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43917 (local address 127.0.97.3:60247)
0929 15:52:13.378222 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:13.378238 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:13.378375 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:13.378741 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:13.378744 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:13.378763 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:52:13.378971 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:13.378977 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:13.380077 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:13.380082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:13.382189 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:13.382198 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:13.383775 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:13.383789 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:13.383979 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:13.384008 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 15:52:13.384095 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 15:52:13.384257 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":77,"thread_start_us":326,"threads_started":1}
I20260929 15:52:13.385355 7619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53842d40c4b949e7bc512667d213f345" candidate_uuid: "6b634f743a52484a9f8f7978e146cbba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f0aa7392be545f89274d4a92743c022" is_pre_election: true
I20260929 15:52:13.385614 7619 raft_consensus.cc:2454] T 53842d40c4b949e7bc512667d213f345 P 2f0aa7392be545f89274d4a92743c022 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b634f743a52484a9f8f7978e146cbba in term 0.
I20260929 15:52:13.385954 7472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53842d40c4b949e7bc512667d213f345" candidate_uuid: "6b634f743a52484a9f8f7978e146cbba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ab5024e8d1a4d259eefb3f421e119fc" is_pre_election: true
I20260929 15:52:13.386142 7697 leader_election.cc:302] T 53842d40c4b949e7bc512667d213f345 P 6b634f743a52484a9f8f7978e146cbba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f0aa7392be545f89274d4a92743c022, 6b634f743a52484a9f8f7978e146cbba; no voters:
I20260929 15:52:13.386379 7472 raft_consensus.cc:2454] T 53842d40c4b949e7bc512667d213f345 P 3ab5024e8d1a4d259eefb3f421e119fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b634f743a52484a9f8f7978e146cbba in term 0.
I20260929 15:52:13.386560 7903 raft_consensus.cc:2781] T 53842d40c4b949e7bc512667d213f345 P 6b634f743a52484a9f8f7978e146cbba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:52:13.386683 7903 raft_consensus.cc:482] T 53842d40c4b949e7bc512667d213f345 P 6b634f743a52484a9f8f7978e146cbba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:52:13.386796 7903 raft_consensus.cc:3037] T 53842d40c4b949e7bc512667d213f345 P 6b634f743a52484a9f8f7978e146cbba [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:52:13.388851 7903 raft_consensus.cc:504] T 53842d40c4b949e7bc512667d213f345 P 6b634f743a52484a9f8f7978e146cbba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0aa7392be545f89274d4a92743c022" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42699 } } peers { permanent_uuid: "3ab5024e8d1a4d259eefb3f421e119fc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43917 } } peers { permanent_uuid: "6b634f743a52484a9f8f7978e146cbba" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35181 } }
I20260929 15:52:13.389544 7903 leader_election.cc:288] T 53842d40c4b949e7bc512667d213f345 P 6b634f743a52484a9f8f7978e146cbba [CANDIDATE]: Term 1 election: Requested vote from peers 2f0aa7392be545f89274d4a92743c022 (127.0.97.2:42699), 3ab5024e8d1a4d259eefb3f421e119fc (127.0.97.1:43917)
I20260929 15:52:13.390025 7619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53842d40c4b949e7bc512667d213f345" candidate_uuid: "6b634f743a52484a9f8f7978e146cbba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f0aa7392be545f89274d4a92743c022"
I20260929 15:52:13.390043 7472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53842d40c4b949e7bc512667d213f345" candidate_uuid: "6b634f743a52484a9f8f7978e146cbba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ab5024e8d1a4d259eefb3f421e119fc"
I20260929 15:52:13.390220 7472 raft_consensus.cc:3037] T 53842d40c4b949e7bc512667d213f345 P 3ab5024e8d1a4d259eefb3f421e119fc [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:52:13.390213 7619 raft_consensus.cc:3037] T 53842d40c4b949e7bc512667d213f345 P 2f0aa7392be545f89274d4a92743c022 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:52:13.392168 7619 raft_consensus.cc:2454] T 53842d40c4b949e7bc512667d213f345 P 2f0aa7392be545f89274d4a92743c022 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b634f743a52484a9f8f7978e146cbba in term 1.
I20260929 15:52:13.392172 7472 raft_consensus.cc:2454] T 53842d40c4b949e7bc512667d213f345 P 3ab5024e8d1a4d259eefb3f421e119fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b634f743a52484a9f8f7978e146cbba in term 1.
I20260929 15:52:13.392547 7697 leader_election.cc:302] T 53842d40c4b949e7bc512667d213f345 P 6b634f743a52484a9f8f7978e146cbba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f0aa7392be545f89274d4a92743c022, 6b634f743a52484a9f8f7978e146cbba; no voters:
I20260929 15:52:13.392813 7903 raft_consensus.cc:2781] T 53842d40c4b949e7bc512667d213f345 P 6b634f743a52484a9f8f7978e146cbba [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:52:13.393227 7903 raft_consensus.cc:686] T 53842d40c4b949e7bc512667d213f345 P 6b634f743a52484a9f8f7978e146cbba [term 1 LEADER]: Becoming Leader. State: Replica: 6b634f743a52484a9f8f7978e146cbba, State: Running, Role: LEADER
I20260929 15:52:13.393638 7903 consensus_queue.cc:237] T 53842d40c4b949e7bc512667d213f345 P 6b634f743a52484a9f8f7978e146cbba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0aa7392be545f89274d4a92743c022" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42699 } } peers { permanent_uuid: "3ab5024e8d1a4d259eefb3f421e119fc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43917 } } peers { permanent_uuid: "6b634f743a52484a9f8f7978e146cbba" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35181 } }
I20260929 15:52:13.394192 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.393801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33067 (local address 127.0.97.3:53455)
0929 15:52:13.393960 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:13.394069 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 15:52:13.397363 7147 catalog_manager.cc:5495] T 53842d40c4b949e7bc512667d213f345 P 6b634f743a52484a9f8f7978e146cbba reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b634f743a52484a9f8f7978e146cbba (127.0.97.3). New cstate: current_term: 1 leader_uuid: "6b634f743a52484a9f8f7978e146cbba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0aa7392be545f89274d4a92743c022" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ab5024e8d1a4d259eefb3f421e119fc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b634f743a52484a9f8f7978e146cbba" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35181 } health_report { overall_health: HEALTHY } } }
I20260929 15:52:13.420501 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.415505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41426 (local address 127.0.97.3:35181)
0929 15:52:13.415667 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:13.415671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:13.415688 (+ 17us) server_negotiation.cc:407] Connection header received
0929 15:52:13.415826 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:13.415831 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:13.415914 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:13.416054 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:52:13.416386 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:13.417118 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:13.418475 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:13.418984 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:13.419087 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:13.419457 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:52:13.420027 (+ 570us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:52:13.420130 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:13.420243 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 15:52:13.420305 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 15:52:13.424297 7619 raft_consensus.cc:1261] T 53842d40c4b949e7bc512667d213f345 P 2f0aa7392be545f89274d4a92743c022 [term 1 FOLLOWER]: Refusing update from remote peer 6b634f743a52484a9f8f7978e146cbba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:52:13.424966 7472 raft_consensus.cc:1261] T 53842d40c4b949e7bc512667d213f345 P 3ab5024e8d1a4d259eefb3f421e119fc [term 1 FOLLOWER]: Refusing update from remote peer 6b634f743a52484a9f8f7978e146cbba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:52:13.425009 7913 consensus_queue.cc:1035] T 53842d40c4b949e7bc512667d213f345 P 6b634f743a52484a9f8f7978e146cbba [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f0aa7392be545f89274d4a92743c022" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:52:13.425602 7903 consensus_queue.cc:1035] T 53842d40c4b949e7bc512667d213f345 P 6b634f743a52484a9f8f7978e146cbba [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ab5024e8d1a4d259eefb3f421e119fc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:52:13.427592 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.427090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33067 (local address 127.0.97.2:53189)
0929 15:52:13.427322 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:13.427406 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260929 15:52:13.428400 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.427955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33067 (local address 127.0.97.1:38339)
0929 15:52:13.428129 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:13.428223 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 15:52:13.431206 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.430784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33067 (local address 127.0.97.3:42897)
0929 15:52:13.430962 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:13.431046 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 15:52:13.435141 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.434682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33067 (local address 127.0.97.2:59711)
0929 15:52:13.434891 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:13.434985 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260929 15:52:13.439767 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.439367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33067 (local address 127.0.97.3:53539)
0929 15:52:13.439523 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:13.439621 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 15:52:13.442026 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.441603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33067 (local address 127.0.97.1:43991)
0929 15:52:13.441770 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:13.441853 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 15:52:13.444475 7147 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:52:13.444729 7147 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:52:13.452140 7147 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2ca5a81736e4a508dea4e247fce6a74: Sending DeleteTablet for 3 replicas of tablet 53842d40c4b949e7bc512667d213f345
I20260929 15:52:13.453107 7452 tablet_service.cc:1514] Processing DeleteTablet for tablet 53842d40c4b949e7bc512667d213f345 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:52:13 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:46368
I20260929 15:52:13.453409 7741 tablet_service.cc:1514] Processing DeleteTablet for tablet 53842d40c4b949e7bc512667d213f345 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:52:13 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:41414
I20260929 15:52:13.453518 7599 tablet_service.cc:1514] Processing DeleteTablet for tablet 53842d40c4b949e7bc512667d213f345 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:52:13 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:48200
I20260929 15:52:13.454996 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 7316
I20260929 15:52:13.466719 7927 tablet_replica.cc:330] T 53842d40c4b949e7bc512667d213f345 P 6b634f743a52484a9f8f7978e146cbba: stopping tablet replica
I20260929 15:52:13.467069 7927 raft_consensus.cc:2229] T 53842d40c4b949e7bc512667d213f345 P 6b634f743a52484a9f8f7978e146cbba [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:52:13.467768 7927 raft_consensus.cc:2258] T 53842d40c4b949e7bc512667d213f345 P 6b634f743a52484a9f8f7978e146cbba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:52:13.468302 7928 tablet_replica.cc:330] T 53842d40c4b949e7bc512667d213f345 P 2f0aa7392be545f89274d4a92743c022: stopping tablet replica
I20260929 15:52:13.468544 7928 raft_consensus.cc:2229] T 53842d40c4b949e7bc512667d213f345 P 2f0aa7392be545f89274d4a92743c022 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:52:13.469851 7927 ts_tablet_manager.cc:1905] T 53842d40c4b949e7bc512667d213f345 P 6b634f743a52484a9f8f7978e146cbba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:52:13.470966 7928 raft_consensus.cc:2258] T 53842d40c4b949e7bc512667d213f345 P 2f0aa7392be545f89274d4a92743c022 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:52:13.473222 7928 ts_tablet_manager.cc:1905] T 53842d40c4b949e7bc512667d213f345 P 2f0aa7392be545f89274d4a92743c022: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:52:13.483512 7926 tablet_replica.cc:330] T 53842d40c4b949e7bc512667d213f345 P 3ab5024e8d1a4d259eefb3f421e119fc: stopping tablet replica
I20260929 15:52:13.483726 7926 raft_consensus.cc:2229] T 53842d40c4b949e7bc512667d213f345 P 3ab5024e8d1a4d259eefb3f421e119fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:52:13.484057 7926 raft_consensus.cc:2258] T 53842d40c4b949e7bc512667d213f345 P 3ab5024e8d1a4d259eefb3f421e119fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:52:13.485234 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.456147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46374 (local address 127.0.97.1:43917)
0929 15:52:13.456369 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:13.456379 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:13.464280 (+ 7901us) server_negotiation.cc:407] Connection header received
0929 15:52:13.464500 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:13.464508 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:13.464628 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:13.464747 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:13.467388 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:13.469126 (+ 1738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:13.470830 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:13.471722 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:13.472258 (+ 536us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:13.476422 (+ 4164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:13.476444 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:13.476471 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:13.476527 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:13.481664 (+ 5137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:13.482835 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:13.482847 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:13.482852 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:13.483035 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:13.483549 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:13.483554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:13.483557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:13.484065 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:13.484374 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:13.484677 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 15:52:13.484891 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 15:52:13.486215 7512 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:52:13.486660 7928 ts_tablet_manager.cc:1918] T 53842d40c4b949e7bc512667d213f345 P 2f0aa7392be545f89274d4a92743c022: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:52:13.486806 7928 log.cc:1196] T 53842d40c4b949e7bc512667d213f345 P 2f0aa7392be545f89274d4a92743c022: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-1/wal/wals/53842d40c4b949e7bc512667d213f345
I20260929 15:52:13.487239 7928 ts_tablet_manager.cc:1939] T 53842d40c4b949e7bc512667d213f345 P 2f0aa7392be545f89274d4a92743c022: Deleting consensus metadata
I20260929 15:52:13.488545 7135 catalog_manager.cc:4841] TS 2f0aa7392be545f89274d4a92743c022 (127.0.97.2:42699): tablet 53842d40c4b949e7bc512667d213f345 (table test-table [id=015f2e180b5046f38193c20cb06b3113]) successfully deleted
I20260929 15:52:13.490628 7927 ts_tablet_manager.cc:1918] T 53842d40c4b949e7bc512667d213f345 P 6b634f743a52484a9f8f7978e146cbba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:52:13.490787 7927 log.cc:1196] T 53842d40c4b949e7bc512667d213f345 P 6b634f743a52484a9f8f7978e146cbba: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697075657932-388-0/minicluster-data/ts-2/wal/wals/53842d40c4b949e7bc512667d213f345
I20260929 15:52:13.491206 7927 ts_tablet_manager.cc:1939] T 53842d40c4b949e7bc512667d213f345 P 6b634f743a52484a9f8f7978e146cbba: Deleting consensus metadata
I20260929 15:52:13.492157 7135 catalog_manager.cc:4841] TS 6b634f743a52484a9f8f7978e146cbba (127.0.97.3:35181): tablet 53842d40c4b949e7bc512667d213f345 (table test-table [id=015f2e180b5046f38193c20cb06b3113]) successfully deleted
I20260929 15:52:13.608212 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 7316
I20260929 15:52:13.630373 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 7529
I20260929 15:52:13.631201 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.630623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43917 (local address 127.0.0.1:46382)
0929 15:52:13.630873 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:13.631003 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260929 15:52:13.631608 7135 catalog_manager.cc:4584] TS 3ab5024e8d1a4d259eefb3f421e119fc (127.0.97.1:43917): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 53842d40c4b949e7bc512667d213f345: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43917: connect: Connection refused (error 111)
I20260929 15:52:13.645716 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.631335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48214 (local address 127.0.97.2:42699)
0929 15:52:13.631520 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:13.631527 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:13.631546 (+ 19us) server_negotiation.cc:407] Connection header received
0929 15:52:13.631607 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:13.631612 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:13.631686 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:13.631783 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:13.633601 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:13.634351 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:13.636008 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:13.636387 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:13.636490 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:13.639692 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:13.639704 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:13.639719 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:13.639748 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:13.642738 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:13.643539 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:13.643545 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:13.643547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:13.643656 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:13.644090 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:13.644094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:13.644096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:13.644452 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:13.644686 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:13.645060 (+ 374us) server_negotiation.cc:299] Negotiation successful
0929 15:52:13.645218 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 15:52:13.646500 7659 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:52:13.653844 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.653373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33067 (local address 127.0.0.1:40816)
0929 15:52:13.653565 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:13.653662 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 15:52:13.654924 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.654541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33067 (local address 127.0.0.1:40826)
0929 15:52:13.654697 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:13.654788 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 15:52:13.692703 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.692166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43917 (local address 127.0.0.1:46398)
0929 15:52:13.692419 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:13.692534 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260929 15:52:13.693828 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.693427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43917 (local address 127.0.0.1:46408)
0929 15:52:13.693592 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:13.693675 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 15:52:13.752395 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 7529
I20260929 15:52:13.762403 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.761848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43917 (local address 127.0.0.1:46414)
0929 15:52:13.762099 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:13.762219 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 15:52:13.763557 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.763156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43917 (local address 127.0.0.1:46426)
0929 15:52:13.763329 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:13.763418 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 15:52:13.770251 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 7677
I20260929 15:52:13.784461 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.771004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41440 (local address 127.0.97.3:35181)
0929 15:52:13.771178 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:13.771186 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:13.771207 (+ 21us) server_negotiation.cc:407] Connection header received
0929 15:52:13.771435 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:13.771442 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:13.771518 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:13.771626 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:13.772993 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:13.773985 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:13.775160 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:13.775604 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:13.775743 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:13.778535 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:13.778555 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:13.778574 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:13.778605 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:13.781671 (+ 3066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:13.782335 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:13.782342 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:13.782344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:13.782479 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:13.782860 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:13.782866 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:13.782869 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:13.783250 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:13.783572 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:13.783813 (+ 241us) server_negotiation.cc:299] Negotiation successful
0929 15:52:13.784029 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 15:52:13.791296 7801 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:52:13.865485 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.864860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43917 (local address 127.0.0.1:46442)
0929 15:52:13.865096 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:13.865267 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260929 15:52:13.866456 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:13.866135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43917 (local address 127.0.0.1:46448)
0929 15:52:13.866283 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:13.866359 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 15:52:13.922494 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 7677
I20260929 15:52:13.943038 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 7012
I20260929 15:52:13.944159 7178 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:52:14.066898 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 7012
I20260929 15:52:14.087739 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 7112
I20260929 15:52:14.088807 7369 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:52:14.203280 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 7112
2026-09-29T15:52:14Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9358 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:52:14 dist-test-slave-r0ck krb5kdc[7938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:52:14 dist-test-slave-r0ck krb5kdc[7938](info): set up 2 sockets
Sep 29 15:52:14 dist-test-slave-r0ck krb5kdc[7938](info): commencing operation
krb5kdc: starting...
W20260929 15:52:15.294369 388 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:52:15 dist-test-slave-r0ck krb5kdc[7938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:52:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:52:15Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260929 15:52:15.506107 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45367
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:52:15.844271 7951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:52:15.844554 7951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:52:15.844712 7951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:52:15.855413 7951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:52:15.855520 7951 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 15:52:15.855612 7951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:52:15.855700 7951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:52:15.855794 7951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:52:15.855881 7951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:52:15.865895 7951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45367
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:52:15.867174 7951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:52:15.868481 7951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:52:15.880394 7957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:15.880353 7956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:15.880419 7959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:52:15.881317 7951 server_base.cc:1034] running on GCE node
I20260929 15:52:15.882148 7951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:52:15.883652 7951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:52:15.884914 7951 hybrid_clock.cc:648] HybridClock initialized: now 1790697135884852 us; error 52 us; skew 500 ppm
I20260929 15:52:15.885246 7951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:52:15 dist-test-slave-r0ck krb5kdc[7938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697135, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:52:15.888587 7951 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260929 15:52:15.890041 7951 webserver.cc:460] Webserver started at http://127.0.97.62:38855/ using document root <none> and password file <none>
I20260929 15:52:15.890692 7951 fs_manager.cc:359] Metadata directory not provided
I20260929 15:52:15.890774 7951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:52:15.891031 7951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:52:15.893949 7951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "035d2b0927e444dc988dd4a5b7e816bd"
format_stamp: "Formatted at 2026-09-29 15:52:15 on dist-test-slave-r0ck"
server_key: "1314ba157ce4102e21eaeb78332b5357"
server_key_iv: "a1b082f92c01c51181cbdf3aadeb862b"
server_key_version: "encryptionkey@0"
I20260929 15:52:15.894695 7951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "035d2b0927e444dc988dd4a5b7e816bd"
format_stamp: "Formatted at 2026-09-29 15:52:15 on dist-test-slave-r0ck"
server_key: "1314ba157ce4102e21eaeb78332b5357"
server_key_iv: "a1b082f92c01c51181cbdf3aadeb862b"
server_key_version: "encryptionkey@0"
I20260929 15:52:15.899732 7951 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 15:52:15.903085 7966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:15.904373 7951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 15:52:15.904541 7951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "035d2b0927e444dc988dd4a5b7e816bd"
format_stamp: "Formatted at 2026-09-29 15:52:15 on dist-test-slave-r0ck"
server_key: "1314ba157ce4102e21eaeb78332b5357"
server_key_iv: "a1b082f92c01c51181cbdf3aadeb862b"
server_key_version: "encryptionkey@0"
I20260929 15:52:15.904747 7951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:52:15.924899 7951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:52:15.928864 7951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:52:15.929131 7951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:52:15.949775 7951 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45367
I20260929 15:52:15.949810 8027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45367 every 8 connection(s)
I20260929 15:52:15.951179 7951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:52:15.954851 8028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:52:15.957501 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 7951
I20260929 15:52:15.957690 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:52:15.958326 388 external_mini_cluster.cc:1428] Setting key 393e903f56ce3a040bc0c1521901797d
I20260929 15:52:15.966627 8028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 035d2b0927e444dc988dd4a5b7e816bd: Bootstrap starting.
I20260929 15:52:15.970379 8028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 035d2b0927e444dc988dd4a5b7e816bd: Neither blocks nor log segments found. Creating new log.
I20260929 15:52:15.971661 8028 log.cc:824] T 00000000000000000000000000000000 P 035d2b0927e444dc988dd4a5b7e816bd: Log is configured to *not* fsync() on all Append() calls
Sep 29 15:52:15 dist-test-slave-r0ck krb5kdc[7938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 15:52:15.975298 8028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 035d2b0927e444dc988dd4a5b7e816bd: No bootstrap required, opened a new log
I20260929 15:52:15.980971 8031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:15.960177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41978 (local address 127.0.97.62:45367)
0929 15:52:15.960891 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:15.960902 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:15.960941 (+ 39us) server_negotiation.cc:407] Connection header received
0929 15:52:15.962700 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:15.962722 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:15.963089 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:15.963407 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:15.965142 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:15.966507 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:15.968233 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:15.968905 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:15.969037 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:15.973889 (+ 4852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:15.973896 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:15.973910 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:15.973943 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:15.977041 (+ 3098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:15.977783 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:15.977794 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:15.977800 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:15.977958 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:15.978334 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:15.978341 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:15.978345 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:15.978833 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:15.979141 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:15.979586 (+ 445us) server_negotiation.cc:299] Negotiation successful
0929 15:52:15.979967 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":337,"threads_started":1}
I20260929 15:52:15.983038 8028 raft_consensus.cc:348] T 00000000000000000000000000000000 P 035d2b0927e444dc988dd4a5b7e816bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "035d2b0927e444dc988dd4a5b7e816bd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45367 } }
I20260929 15:52:15.983392 8028 raft_consensus.cc:374] T 00000000000000000000000000000000 P 035d2b0927e444dc988dd4a5b7e816bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:52:15.983528 8028 raft_consensus.cc:729] T 00000000000000000000000000000000 P 035d2b0927e444dc988dd4a5b7e816bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 035d2b0927e444dc988dd4a5b7e816bd, State: Initialized, Role: FOLLOWER
I20260929 15:52:15.984124 8028 consensus_queue.cc:260] T 00000000000000000000000000000000 P 035d2b0927e444dc988dd4a5b7e816bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "035d2b0927e444dc988dd4a5b7e816bd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45367 } }
I20260929 15:52:15.984328 8028 raft_consensus.cc:388] T 00000000000000000000000000000000 P 035d2b0927e444dc988dd4a5b7e816bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:52:15.984419 8028 raft_consensus.cc:482] T 00000000000000000000000000000000 P 035d2b0927e444dc988dd4a5b7e816bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:52:15.984531 8028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 035d2b0927e444dc988dd4a5b7e816bd [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:52:15.986863 8028 raft_consensus.cc:504] T 00000000000000000000000000000000 P 035d2b0927e444dc988dd4a5b7e816bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "035d2b0927e444dc988dd4a5b7e816bd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45367 } }
I20260929 15:52:15.987313 8028 leader_election.cc:302] T 00000000000000000000000000000000 P 035d2b0927e444dc988dd4a5b7e816bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 035d2b0927e444dc988dd4a5b7e816bd; no voters:
I20260929 15:52:15.987775 8028 leader_election.cc:288] T 00000000000000000000000000000000 P 035d2b0927e444dc988dd4a5b7e816bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:52:15.987921 8033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 035d2b0927e444dc988dd4a5b7e816bd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:52:15.988595 8033 raft_consensus.cc:686] T 00000000000000000000000000000000 P 035d2b0927e444dc988dd4a5b7e816bd [term 1 LEADER]: Becoming Leader. State: Replica: 035d2b0927e444dc988dd4a5b7e816bd, State: Running, Role: LEADER
I20260929 15:52:15.989149 8033 consensus_queue.cc:237] T 00000000000000000000000000000000 P 035d2b0927e444dc988dd4a5b7e816bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "035d2b0927e444dc988dd4a5b7e816bd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45367 } }
I20260929 15:52:15.989835 8028 sys_catalog.cc:564] T 00000000000000000000000000000000 P 035d2b0927e444dc988dd4a5b7e816bd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:52:15.991590 8034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 035d2b0927e444dc988dd4a5b7e816bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "035d2b0927e444dc988dd4a5b7e816bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "035d2b0927e444dc988dd4a5b7e816bd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45367 } } }
I20260929 15:52:15.991953 8034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 035d2b0927e444dc988dd4a5b7e816bd [sys.catalog]: This master's current role is: LEADER
I20260929 15:52:15.993177 8035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 035d2b0927e444dc988dd4a5b7e816bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 035d2b0927e444dc988dd4a5b7e816bd. Latest consensus state: current_term: 1 leader_uuid: "035d2b0927e444dc988dd4a5b7e816bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "035d2b0927e444dc988dd4a5b7e816bd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45367 } } }
I20260929 15:52:15.993619 8035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 035d2b0927e444dc988dd4a5b7e816bd [sys.catalog]: This master's current role is: LEADER
I20260929 15:52:15.995353 8039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:52:16.000598 8039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:52:16.010329 8039 catalog_manager.cc:1347] Generated new cluster ID: b5ee141e63c04232bbe8fdeb84b4e473
I20260929 15:52:16.010479 8039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:52:16.021997 8039 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:52:16.023423 8039 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:52:16.032832 8039 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 035d2b0927e444dc988dd4a5b7e816bd: Generated new TSK 0
I20260929 15:52:16.033757 8039 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260929 15:52:16.101518 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45367
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:52:16.424187 8054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:52:16.424471 8054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:52:16.424638 8054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:52:16.431870 8054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:52:16.431974 8054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:52:16.432166 8054 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 15:52:16.432279 8054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:52:16.441813 8054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:45367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:52:16.443066 8054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:52:16.444334 8054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:52:16.456784 8060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:16.456812 8062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:16.457302 8059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:52:16.457661 8054 server_base.cc:1034] running on GCE node
I20260929 15:52:16.458168 8054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:52:16.458984 8054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:52:16.460222 8054 hybrid_clock.cc:648] HybridClock initialized: now 1790697136460150 us; error 54 us; skew 500 ppm
I20260929 15:52:16.460533 8054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:52:16 dist-test-slave-r0ck krb5kdc[7938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697136, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:52:16.463584 8054 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260929 15:52:16.464893 8054 webserver.cc:460] Webserver started at http://127.0.97.1:42321/ using document root <none> and password file <none>
I20260929 15:52:16.465631 8054 fs_manager.cc:359] Metadata directory not provided
I20260929 15:52:16.465756 8054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:52:16.466068 8054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:52:16.468848 8054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-0/data/instance:
uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96"
format_stamp: "Formatted at 2026-09-29 15:52:16 on dist-test-slave-r0ck"
server_key: "a17d307aec90bc40af7c5d3e4bcb3178"
server_key_iv: "74bc15d36ad8da67a7cde1ef1c8dfc50"
server_key_version: "encryptionkey@0"
I20260929 15:52:16.469697 8054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96"
format_stamp: "Formatted at 2026-09-29 15:52:16 on dist-test-slave-r0ck"
server_key: "a17d307aec90bc40af7c5d3e4bcb3178"
server_key_iv: "74bc15d36ad8da67a7cde1ef1c8dfc50"
server_key_version: "encryptionkey@0"
I20260929 15:52:16.474552 8054 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 15:52:16.477806 8069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:16.479146 8054 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 15:52:16.479328 8054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-0/wal
uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96"
format_stamp: "Formatted at 2026-09-29 15:52:16 on dist-test-slave-r0ck"
server_key: "a17d307aec90bc40af7c5d3e4bcb3178"
server_key_iv: "74bc15d36ad8da67a7cde1ef1c8dfc50"
server_key_version: "encryptionkey@0"
I20260929 15:52:16.479573 8054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:52:16.498039 8054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:52:16.501919 8054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:52:16.502175 8054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:52:16.504803 8054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:52:16.504905 8054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:16.505064 8054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:52:16.505141 8054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 15:52:16 dist-test-slave-r0ck krb5kdc[7938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697136, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 15:52:16.528173 8077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:16.505222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58907 (local address 127.0.97.62:45367)
0929 15:52:16.506013 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:16.506020 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:16.506927 (+ 907us) server_negotiation.cc:407] Connection header received
0929 15:52:16.509094 (+ 2167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:16.509103 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:16.509206 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:16.509370 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:16.512237 (+ 2867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:16.513185 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:16.516216 (+ 3031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:16.516624 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:16.516720 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:16.521255 (+ 4535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:16.521264 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:16.521271 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:16.521315 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:16.524468 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:16.525517 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:16.525524 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:16.525528 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:16.525670 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:16.526200 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:16.526208 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:16.526214 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:16.526607 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:16.526787 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:16.527609 (+ 822us) server_negotiation.cc:299] Negotiation successful
0929 15:52:16.527812 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":575,"threads_started":1}
I20260929 15:52:16.529597 8079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:16.505841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45367 (local address 127.0.97.1:58907)
0929 15:52:16.506751 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:16.506805 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:16.508816 (+ 2011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:16.509852 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:16.509867 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:16.511042 (+ 1175us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:16.512024 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:16.512059 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:16.513378 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:16.513385 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:16.516045 (+ 2660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:16.516064 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:16.516878 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:16.516888 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:16.517232 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:16.518025 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:16.518062 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:16.521023 (+ 2961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:16.524622 (+ 3599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:16.524630 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:16.524648 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:16.525291 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:16.525764 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:16.525772 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:16.525776 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:16.526056 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:16.526773 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:16.526783 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:16.527168 (+ 385us) client_negotiation.cc:769] Sending connection context
0929 15:52:16.527524 (+ 356us) client_negotiation.cc:240] Negotiation successful
0929 15:52:16.528036 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":423,"threads_started":1}
I20260929 15:52:16.551894 8077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:16.536536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59255 (local address 127.0.97.62:45367)
0929 15:52:16.536732 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:16.536740 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:16.537432 (+ 692us) server_negotiation.cc:407] Connection header received
0929 15:52:16.537728 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:16.537733 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:16.537802 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:16.537903 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:16.539542 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:16.540786 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:16.543471 (+ 2685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:16.543835 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:16.543933 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:16.546713 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:16.546718 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:16.546721 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:16.546748 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:16.549218 (+ 2470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:16.550148 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:16.550152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:16.550154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:16.550261 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:16.550660 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:16.550663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:16.550665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:16.550897 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:16.551006 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:16.551501 (+ 495us) server_negotiation.cc:299] Negotiation successful
0929 15:52:16.551650 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 15:52:16.552529 8144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:16.536536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45367 (local address 127.0.97.1:59255)
0929 15:52:16.537313 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:16.537333 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:16.537560 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:16.537927 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:16.537932 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:16.538742 (+ 810us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:16.539332 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:16.539355 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:16.541036 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:16.541044 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:16.543345 (+ 2301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:16.543358 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:16.544018 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:16.544025 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:16.544209 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:16.545080 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:16.545106 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:16.546517 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:16.549368 (+ 2851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:16.549375 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:16.549382 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:16.550025 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:16.550366 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:16.550371 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:16.550376 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:16.550552 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:16.551006 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:16.551013 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:16.551258 (+ 245us) client_negotiation.cc:769] Sending connection context
0929 15:52:16.551423 (+ 165us) client_negotiation.cc:240] Negotiation successful
0929 15:52:16.551835 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":420,"threads_started":1}
I20260929 15:52:16.559204 8054 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34987
I20260929 15:52:16.559239 8189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34987 every 8 connection(s)
I20260929 15:52:16.560354 8054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:52:16.562259 8190 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45367
I20260929 15:52:16.562476 8190 heartbeater.cc:464] Registering TS with master...
I20260929 15:52:16.563056 8190 heartbeater.cc:511] Master 127.0.97.62:45367 requested a full tablet report, sending...
I20260929 15:52:16.563817 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 8054
I20260929 15:52:16.564030 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:52:16.564620 388 external_mini_cluster.cc:1428] Setting key 8b571a50c6ba966a8556771461e11b52
I20260929 15:52:16.564935 7983 ts_manager.cc:194] Registered new tserver with Master: 8bc1e5f5f2da4191ab74dbdb9e29af96 (127.0.97.1:34987)
I20260929 15:52:16.566956 7983 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58907
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260929 15:52:16.608796 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45367
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:52:16.920436 8194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:52:16.920730 8194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:52:16.920898 8194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:52:16.927891 8194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:52:16.928027 8194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:52:16.928177 8194 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 15:52:16.928314 8194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:52:16.937808 8194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:45367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:52:16.938973 8194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:52:16.940235 8194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:52:16.953301 8199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:16.954099 8200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:16.954885 8202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:52:16.955869 8194 server_base.cc:1034] running on GCE node
I20260929 15:52:16.956489 8194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:52:16.957422 8194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:52:16.958649 8194 hybrid_clock.cc:648] HybridClock initialized: now 1790697136958599 us; error 25 us; skew 500 ppm
I20260929 15:52:16.958946 8194 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:52:16 dist-test-slave-r0ck krb5kdc[7938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697136, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:52:16.965186 8194 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260929 15:52:16.966645 8194 webserver.cc:460] Webserver started at http://127.0.97.2:45471/ using document root <none> and password file <none>
I20260929 15:52:16.967316 8194 fs_manager.cc:359] Metadata directory not provided
I20260929 15:52:16.967423 8194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:52:16.967731 8194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:52:16.970498 8194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-1/data/instance:
uuid: "b76aeb70d77b4a39b3243601bca4ffcb"
format_stamp: "Formatted at 2026-09-29 15:52:16 on dist-test-slave-r0ck"
server_key: "806cecc8782c163a53dc220a7efc09a5"
server_key_iv: "ba11f8d695c1142ba281b576dcfd7d4c"
server_key_version: "encryptionkey@0"
I20260929 15:52:16.971247 8194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b76aeb70d77b4a39b3243601bca4ffcb"
format_stamp: "Formatted at 2026-09-29 15:52:16 on dist-test-slave-r0ck"
server_key: "806cecc8782c163a53dc220a7efc09a5"
server_key_iv: "ba11f8d695c1142ba281b576dcfd7d4c"
server_key_version: "encryptionkey@0"
I20260929 15:52:16.976011 8194 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 15:52:16.979334 8209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:16.980564 8194 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 15:52:16.980726 8194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-1/wal
uuid: "b76aeb70d77b4a39b3243601bca4ffcb"
format_stamp: "Formatted at 2026-09-29 15:52:16 on dist-test-slave-r0ck"
server_key: "806cecc8782c163a53dc220a7efc09a5"
server_key_iv: "ba11f8d695c1142ba281b576dcfd7d4c"
server_key_version: "encryptionkey@0"
I20260929 15:52:16.980950 8194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:52:16.993582 8194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:52:16.997835 8194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:52:16.998097 8194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:52:17.000488 8194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:52:17.000595 8194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:17.000735 8194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:52:17.000823 8194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 15:52:17 dist-test-slave-r0ck krb5kdc[7938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697136, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 15:52:17.027320 8077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:17.001311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42171 (local address 127.0.97.62:45367)
0929 15:52:17.001536 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:17.001542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:17.002838 (+ 1296us) server_negotiation.cc:407] Connection header received
0929 15:52:17.005989 (+ 3151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:17.005999 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:17.006121 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:17.006198 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:17.009428 (+ 3230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.010434 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.013966 (+ 3532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.014413 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.014517 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:17.020058 (+ 5541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:17.020067 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:17.020075 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:17.020131 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:17.023614 (+ 3483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:17.024568 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:17.024574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:17.024577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:17.024717 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:17.025260 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:17.025265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:17.025269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:17.025663 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:17.025835 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:17.026748 (+ 913us) server_negotiation.cc:299] Negotiation successful
0929 15:52:17.026972 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260929 15:52:17.028363 8218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:17.001786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45367 (local address 127.0.97.2:42171)
0929 15:52:17.002645 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:17.002704 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:17.005593 (+ 2889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:17.006862 (+ 1269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:17.006879 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:17.007950 (+ 1071us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:17.009158 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:17.009195 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.010615 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.010623 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:17.013774 (+ 3151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:17.013794 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.014734 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.014745 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:17.015149 (+ 404us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:17.016043 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:17.016086 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:17.019750 (+ 3664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:17.023755 (+ 4005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:17.023764 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:17.023785 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:17.024447 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:17.024874 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:17.024879 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:17.024886 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:17.025132 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:17.025829 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:17.025837 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:17.026236 (+ 399us) client_negotiation.cc:769] Sending connection context
0929 15:52:17.026651 (+ 415us) client_negotiation.cc:240] Negotiation successful
0929 15:52:17.027186 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":331,"threads_started":1}
I20260929 15:52:17.052090 8077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:17.036001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41711 (local address 127.0.97.62:45367)
0929 15:52:17.036229 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:17.036238 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:17.037219 (+ 981us) server_negotiation.cc:407] Connection header received
0929 15:52:17.037455 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:17.037462 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:17.037543 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:17.037622 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:17.039132 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.040399 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.042300 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.043002 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.043138 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:17.046013 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:17.046018 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:17.046021 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:17.046061 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:17.049562 (+ 3501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:17.050215 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:17.050221 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:17.050224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:17.050342 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:17.050700 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:17.050705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:17.050708 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:17.051021 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:17.051166 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:17.051551 (+ 385us) server_negotiation.cc:299] Negotiation successful
0929 15:52:17.051756 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 15:52:17.052240 8290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:17.035998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45367 (local address 127.0.97.2:41711)
0929 15:52:17.037087 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:17.037112 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:17.037297 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:17.037690 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:17.037698 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:17.038452 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:17.038916 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:17.038930 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.040566 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.040572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:17.042153 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:17.042164 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.043424 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.043434 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:17.043705 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:17.044612 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:17.044649 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:17.045823 (+ 1174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:17.049688 (+ 3865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:17.049693 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:17.049697 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:17.050092 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:17.050456 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:17.050462 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:17.050466 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:17.050589 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:17.051142 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:17.051147 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:17.051317 (+ 170us) client_negotiation.cc:769] Sending connection context
0929 15:52:17.051457 (+ 140us) client_negotiation.cc:240] Negotiation successful
0929 15:52:17.051771 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":906,"thread_start_us":655,"threads_started":1}
I20260929 15:52:17.057101 8194 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38887
I20260929 15:52:17.057127 8328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38887 every 8 connection(s)
I20260929 15:52:17.058284 8194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:52:17.059832 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 8194
I20260929 15:52:17.060014 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:52:17.060009 8329 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45367
I20260929 15:52:17.060181 8329 heartbeater.cc:464] Registering TS with master...
I20260929 15:52:17.060508 388 external_mini_cluster.cc:1428] Setting key aa46c6e252063c1079f6082054d6238f
I20260929 15:52:17.060722 8329 heartbeater.cc:511] Master 127.0.97.62:45367 requested a full tablet report, sending...
I20260929 15:52:17.061947 7982 ts_manager.cc:194] Registered new tserver with Master: b76aeb70d77b4a39b3243601bca4ffcb (127.0.97.2:38887)
I20260929 15:52:17.062742 7982 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42171
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260929 15:52:17.104615 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45367
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:52:17.413853 8333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:52:17.414113 8333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:52:17.414268 8333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:52:17.421308 8333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:52:17.421479 8333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:52:17.421639 8333 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 15:52:17.421759 8333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:52:17.431313 8333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:45367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:52:17.432590 8333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:52:17.434029 8333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:52:17.446447 8338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:17.446530 8339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:17.446493 8341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:52:17.447414 8333 server_base.cc:1034] running on GCE node
I20260929 15:52:17.447965 8333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:52:17.448923 8333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:52:17.450296 8333 hybrid_clock.cc:648] HybridClock initialized: now 1790697137450168 us; error 49 us; skew 500 ppm
I20260929 15:52:17.450829 8333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:52:17 dist-test-slave-r0ck krb5kdc[7938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697137, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:52:17.454797 8333 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260929 15:52:17.456393 8333 webserver.cc:460] Webserver started at http://127.0.97.3:43149/ using document root <none> and password file <none>
I20260929 15:52:17.457115 8333 fs_manager.cc:359] Metadata directory not provided
I20260929 15:52:17.457250 8333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:52:17.457619 8333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:52:17.460448 8333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-2/data/instance:
uuid: "3a94f1c6bf464b549de59a4a1aecd307"
format_stamp: "Formatted at 2026-09-29 15:52:17 on dist-test-slave-r0ck"
server_key: "a28b307820d48c22489fa61b7e7e7bec"
server_key_iv: "8e49e00df9944a5e15f18fbc23465a41"
server_key_version: "encryptionkey@0"
I20260929 15:52:17.461249 8333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "3a94f1c6bf464b549de59a4a1aecd307"
format_stamp: "Formatted at 2026-09-29 15:52:17 on dist-test-slave-r0ck"
server_key: "a28b307820d48c22489fa61b7e7e7bec"
server_key_iv: "8e49e00df9944a5e15f18fbc23465a41"
server_key_version: "encryptionkey@0"
I20260929 15:52:17.466305 8333 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 15:52:17.469748 8348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:17.471086 8333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 15:52:17.471261 8333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-2/wal
uuid: "3a94f1c6bf464b549de59a4a1aecd307"
format_stamp: "Formatted at 2026-09-29 15:52:17 on dist-test-slave-r0ck"
server_key: "a28b307820d48c22489fa61b7e7e7bec"
server_key_iv: "8e49e00df9944a5e15f18fbc23465a41"
server_key_version: "encryptionkey@0"
I20260929 15:52:17.471488 8333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:52:17.481948 8333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:52:17.486349 8333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:52:17.486611 8333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:52:17.489077 8333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:52:17.489182 8333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:17.489388 8333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:52:17.489488 8333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Sep 29 15:52:17 dist-test-slave-r0ck krb5kdc[7938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697137, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 15:52:17.512125 8077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:17.489558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37543 (local address 127.0.97.62:45367)
0929 15:52:17.489797 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:17.489807 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:17.491123 (+ 1316us) server_negotiation.cc:407] Connection header received
0929 15:52:17.493727 (+ 2604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:17.493736 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:17.493821 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:17.493890 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:17.496179 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.497205 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.499847 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.500439 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.500577 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:17.504953 (+ 4376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:17.504958 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:17.504961 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:17.504991 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:17.508836 (+ 3845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:17.509917 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:17.509922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:17.509924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:17.510038 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:17.510472 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:17.510477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:17.510481 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:17.510745 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:17.510897 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:17.511583 (+ 686us) server_negotiation.cc:299] Negotiation successful
0929 15:52:17.511787 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 15:52:17.513051 8357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:17.490069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45367 (local address 127.0.97.3:37543)
0929 15:52:17.490966 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:17.491018 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:17.493374 (+ 2356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:17.494239 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:17.494246 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:17.495032 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:17.496009 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:17.496034 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.497381 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.497392 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:17.499706 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:17.499717 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.500663 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.500671 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:17.500883 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:17.501580 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:17.501605 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:17.504727 (+ 3122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:17.508991 (+ 4264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:17.509000 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:17.509017 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:17.509758 (+ 741us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:17.510147 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:17.510154 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:17.510157 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:17.510348 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:17.510877 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:17.510885 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:17.511200 (+ 315us) client_negotiation.cc:769] Sending connection context
0929 15:52:17.511445 (+ 245us) client_negotiation.cc:240] Negotiation successful
0929 15:52:17.511932 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":353,"threads_started":1}
I20260929 15:52:17.532248 8077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:17.518535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41857 (local address 127.0.97.62:45367)
0929 15:52:17.518880 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:17.518884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:17.519404 (+ 520us) server_negotiation.cc:407] Connection header received
0929 15:52:17.519594 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:17.519597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:17.519649 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:17.519723 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:17.521111 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.522083 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.524122 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.524549 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.524679 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:17.526534 (+ 1855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:17.526539 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:17.526542 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:17.526568 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:17.529559 (+ 2991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:17.530302 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:17.530314 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:17.530317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:17.530424 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:17.530773 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:17.530780 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:17.530783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:17.531059 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:17.531230 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:17.531645 (+ 415us) server_negotiation.cc:299] Negotiation successful
0929 15:52:17.531842 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252}
I20260929 15:52:17.532312 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:17.518596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45367 (local address 127.0.97.3:41857)
0929 15:52:17.519279 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:17.519305 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:17.519481 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:17.519787 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:17.519795 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:17.520480 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:17.520926 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:17.520942 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.522238 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.522243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:17.523976 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:17.523988 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.524740 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.524747 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:17.524917 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:17.525668 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:17.525685 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:17.526331 (+ 646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:17.529682 (+ 3351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:17.529687 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:17.529690 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:17.530181 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:17.530539 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:17.530547 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:17.530551 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:17.530672 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:17.531197 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:17.531203 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:17.531379 (+ 176us) client_negotiation.cc:769] Sending connection context
0929 15:52:17.531536 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 15:52:17.531830 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":334,"threads_started":1}
I20260929 15:52:17.546528 8333 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34959
I20260929 15:52:17.546561 8467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34959 every 8 connection(s)
I20260929 15:52:17.547746 8333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:52:17.549831 8468 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45367
I20260929 15:52:17.550011 8468 heartbeater.cc:464] Registering TS with master...
I20260929 15:52:17.550565 8468 heartbeater.cc:511] Master 127.0.97.62:45367 requested a full tablet report, sending...
I20260929 15:52:17.551720 7982 ts_manager.cc:194] Registered new tserver with Master: 3a94f1c6bf464b549de59a4a1aecd307 (127.0.97.3:34959)
I20260929 15:52:17.552570 7982 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:37543
I20260929 15:52:17.553951 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 8333
I20260929 15:52:17.554102 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:52:17.554526 388 external_mini_cluster.cc:1428] Setting key 88a11a520afea60862b58c31545451c6
I20260929 15:52:17.557129 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:52:17.569453 8190 heartbeater.cc:503] Master 127.0.97.62:45367 was elected leader, sending a full tablet report...
Sep 29 15:52:17 dist-test-slave-r0ck krb5kdc[7938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697137, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 15:52:17 dist-test-slave-r0ck krb5kdc[7938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:17 dist-test-slave-r0ck krb5kdc[7938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 15:52:17.603080 8077 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:17.593228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60780 (local address 127.0.97.62:45367)
0929 15:52:17.593468 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:17.593474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:17.593676 (+ 202us) server_negotiation.cc:407] Connection header received
0929 15:52:17.593815 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:17.593819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:17.593884 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:17.593979 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:17.595251 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.595977 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.597180 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.597667 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.597761 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:17.602806 (+ 5045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60780: BlockingRecv error: recv got EOF from 127.0.0.1:60780 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 29 15:52:17 dist-test-slave-r0ck krb5kdc[7938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697137, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 15:52:17.618081 8077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:17.605869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60796 (local address 127.0.97.62:45367)
0929 15:52:17.606075 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:17.606080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:17.606304 (+ 224us) server_negotiation.cc:407] Connection header received
0929 15:52:17.606547 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:17.606551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:17.606616 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:17.606693 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:17.607726 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.608489 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.609799 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.610191 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.610284 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:17.613565 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:17.613570 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:17.613573 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:17.613600 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:17.616033 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:17.616603 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:17.616607 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:17.616610 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:17.616727 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:17.617035 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:17.617038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:17.617040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:17.617279 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:17.617568 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:17.617715 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 15:52:17.617860 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 15:52:17.621755 7982 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:52:17.628319 7982 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:52:17.650816 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:17.641421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38887 (local address 127.0.0.1:48030)
0929 15:52:17.641956 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:17.641976 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:17.642078 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:17.643471 (+ 1393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:17.643477 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:17.643508 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:52:17.643939 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:17.643950 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.646074 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.646079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:17.648358 (+ 2279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:17.648370 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.649992 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.649998 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:17.650245 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:17.650274 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 15:52:17.650354 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 15:52:17.650541 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":71,"thread_start_us":298,"threads_started":1}
I20260929 15:52:17.650871 8489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:17.641752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48030 (local address 127.0.97.2:38887)
0929 15:52:17.642912 (+ 1160us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:17.642921 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:17.642948 (+ 27us) server_negotiation.cc:407] Connection header received
0929 15:52:17.643043 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:17.643050 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:17.643266 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:17.644118 (+ 852us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:52:17.644313 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.645620 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.648497 (+ 2877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.649759 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.650168 (+ 409us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:17.650343 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:17.650408 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 15:52:17.650563 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":993,"thread_start_us":877,"threads_started":1}
I20260929 15:52:17.650879 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:17.642771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39152 (local address 127.0.97.1:34987)
0929 15:52:17.643709 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:17.643720 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:17.643747 (+ 27us) server_negotiation.cc:407] Connection header received
0929 15:52:17.643845 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:17.643853 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:17.644064 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:17.644243 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:52:17.644821 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.646204 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.648235 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.649722 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.649865 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:17.650084 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:17.650474 (+ 390us) server_negotiation.cc:299] Negotiation successful
0929 15:52:17.650607 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":799,"thread_start_us":677,"threads_started":1}
I20260929 15:52:17.651664 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:17.640149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34959 (local address 127.0.0.1:49876)
0929 15:52:17.641021 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:17.641071 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:17.641224 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:17.643262 (+ 2038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:17.643269 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:17.643298 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:52:17.643763 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:17.643773 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.645667 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.645672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:17.648259 (+ 2587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:17.648273 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.650042 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.650049 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:17.651126 (+ 1077us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:17.651153 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 15:52:17.651238 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 15:52:17.651362 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":887,"thread_start_us":496,"threads_started":1}
I20260929 15:52:17.651813 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:17.642319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34987 (local address 127.0.0.1:39152)
0929 15:52:17.643118 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:17.643139 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:17.643243 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:17.644254 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:17.644260 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:17.644285 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:52:17.644636 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:17.644649 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.646369 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.646375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:17.648068 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:17.648079 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.649927 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.649934 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:17.650151 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:17.650205 (+ 54us) client_negotiation.cc:769] Sending connection context
0929 15:52:17.650328 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 15:52:17.651438 (+ 1110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"mutex_wait_us":968,"thread_start_us":609,"threads_started":1}
I20260929 15:52:17.652426 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:17.640580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49876 (local address 127.0.97.3:34959)
0929 15:52:17.642664 (+ 2084us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:17.642675 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:17.642707 (+ 32us) server_negotiation.cc:407] Connection header received
0929 15:52:17.642794 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:17.642803 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:17.643047 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:17.643943 (+ 896us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:52:17.644162 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.645530 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.648412 (+ 2882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.649799 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.650229 (+ 430us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:17.650444 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:17.651906 (+ 1462us) server_negotiation.cc:299] Negotiation successful
0929 15:52:17.652104 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1909,"thread_start_us":1706,"threads_started":1}
I20260929 15:52:17.654704 8259 tablet_service.cc:1467] Processing CreateTablet for tablet 14818eb652a8457a9bfff03870784ee6 (DEFAULT_TABLE table=test-table [id=370adf68136c441aa27e9eeaccbdc432]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:52:17.655136 8120 tablet_service.cc:1467] Processing CreateTablet for tablet 14818eb652a8457a9bfff03870784ee6 (DEFAULT_TABLE table=test-table [id=370adf68136c441aa27e9eeaccbdc432]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:52:17.656008 8259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14818eb652a8457a9bfff03870784ee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:52:17.655637 8398 tablet_service.cc:1467] Processing CreateTablet for tablet 14818eb652a8457a9bfff03870784ee6 (DEFAULT_TABLE table=test-table [id=370adf68136c441aa27e9eeaccbdc432]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:52:17.656430 8120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14818eb652a8457a9bfff03870784ee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:52:17.656903 8398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14818eb652a8457a9bfff03870784ee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:52:17.667178 8492 tablet_bootstrap.cc:492] T 14818eb652a8457a9bfff03870784ee6 P b76aeb70d77b4a39b3243601bca4ffcb: Bootstrap starting.
I20260929 15:52:17.670317 8494 tablet_bootstrap.cc:492] T 14818eb652a8457a9bfff03870784ee6 P 3a94f1c6bf464b549de59a4a1aecd307: Bootstrap starting.
I20260929 15:52:17.670804 8492 tablet_bootstrap.cc:654] T 14818eb652a8457a9bfff03870784ee6 P b76aeb70d77b4a39b3243601bca4ffcb: Neither blocks nor log segments found. Creating new log.
I20260929 15:52:17.672434 8492 log.cc:824] T 14818eb652a8457a9bfff03870784ee6 P b76aeb70d77b4a39b3243601bca4ffcb: Log is configured to *not* fsync() on all Append() calls
I20260929 15:52:17.673673 8494 tablet_bootstrap.cc:654] T 14818eb652a8457a9bfff03870784ee6 P 3a94f1c6bf464b549de59a4a1aecd307: Neither blocks nor log segments found. Creating new log.
I20260929 15:52:17.673704 8493 tablet_bootstrap.cc:492] T 14818eb652a8457a9bfff03870784ee6 P 8bc1e5f5f2da4191ab74dbdb9e29af96: Bootstrap starting.
I20260929 15:52:17.674847 8494 log.cc:824] T 14818eb652a8457a9bfff03870784ee6 P 3a94f1c6bf464b549de59a4a1aecd307: Log is configured to *not* fsync() on all Append() calls
I20260929 15:52:17.675737 8492 tablet_bootstrap.cc:492] T 14818eb652a8457a9bfff03870784ee6 P b76aeb70d77b4a39b3243601bca4ffcb: No bootstrap required, opened a new log
I20260929 15:52:17.676007 8492 ts_tablet_manager.cc:1397] T 14818eb652a8457a9bfff03870784ee6 P b76aeb70d77b4a39b3243601bca4ffcb: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260929 15:52:17.677026 8493 tablet_bootstrap.cc:654] T 14818eb652a8457a9bfff03870784ee6 P 8bc1e5f5f2da4191ab74dbdb9e29af96: Neither blocks nor log segments found. Creating new log.
I20260929 15:52:17.677826 8494 tablet_bootstrap.cc:492] T 14818eb652a8457a9bfff03870784ee6 P 3a94f1c6bf464b549de59a4a1aecd307: No bootstrap required, opened a new log
I20260929 15:52:17.678069 8494 ts_tablet_manager.cc:1397] T 14818eb652a8457a9bfff03870784ee6 P 3a94f1c6bf464b549de59a4a1aecd307: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 15:52:17.678236 8493 log.cc:824] T 14818eb652a8457a9bfff03870784ee6 P 8bc1e5f5f2da4191ab74dbdb9e29af96: Log is configured to *not* fsync() on all Append() calls
I20260929 15:52:17.680940 8493 tablet_bootstrap.cc:492] T 14818eb652a8457a9bfff03870784ee6 P 8bc1e5f5f2da4191ab74dbdb9e29af96: No bootstrap required, opened a new log
I20260929 15:52:17.681170 8493 ts_tablet_manager.cc:1397] T 14818eb652a8457a9bfff03870784ee6 P 8bc1e5f5f2da4191ab74dbdb9e29af96: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260929 15:52:17.684461 8492 raft_consensus.cc:348] T 14818eb652a8457a9bfff03870784ee6 P b76aeb70d77b4a39b3243601bca4ffcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } }
I20260929 15:52:17.684794 8492 raft_consensus.cc:374] T 14818eb652a8457a9bfff03870784ee6 P b76aeb70d77b4a39b3243601bca4ffcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:52:17.684887 8492 raft_consensus.cc:729] T 14818eb652a8457a9bfff03870784ee6 P b76aeb70d77b4a39b3243601bca4ffcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b76aeb70d77b4a39b3243601bca4ffcb, State: Initialized, Role: FOLLOWER
I20260929 15:52:17.685510 8492 consensus_queue.cc:260] T 14818eb652a8457a9bfff03870784ee6 P b76aeb70d77b4a39b3243601bca4ffcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } }
I20260929 15:52:17.688267 8494 raft_consensus.cc:348] T 14818eb652a8457a9bfff03870784ee6 P 3a94f1c6bf464b549de59a4a1aecd307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } }
I20260929 15:52:17.688663 8494 raft_consensus.cc:374] T 14818eb652a8457a9bfff03870784ee6 P 3a94f1c6bf464b549de59a4a1aecd307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:52:17.688774 8494 raft_consensus.cc:729] T 14818eb652a8457a9bfff03870784ee6 P 3a94f1c6bf464b549de59a4a1aecd307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a94f1c6bf464b549de59a4a1aecd307, State: Initialized, Role: FOLLOWER
I20260929 15:52:17.689440 8329 heartbeater.cc:503] Master 127.0.97.62:45367 was elected leader, sending a full tablet report...
I20260929 15:52:17.690088 8492 ts_tablet_manager.cc:1428] T 14818eb652a8457a9bfff03870784ee6 P b76aeb70d77b4a39b3243601bca4ffcb: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260929 15:52:17.690768 8494 consensus_queue.cc:260] T 14818eb652a8457a9bfff03870784ee6 P 3a94f1c6bf464b549de59a4a1aecd307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } }
I20260929 15:52:17.691316 8493 raft_consensus.cc:348] T 14818eb652a8457a9bfff03870784ee6 P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } }
I20260929 15:52:17.691800 8493 raft_consensus.cc:374] T 14818eb652a8457a9bfff03870784ee6 P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:52:17.691928 8493 raft_consensus.cc:729] T 14818eb652a8457a9bfff03870784ee6 P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bc1e5f5f2da4191ab74dbdb9e29af96, State: Initialized, Role: FOLLOWER
I20260929 15:52:17.692641 8493 consensus_queue.cc:260] T 14818eb652a8457a9bfff03870784ee6 P 8bc1e5f5f2da4191ab74dbdb9e29af96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } }
I20260929 15:52:17.693794 8468 heartbeater.cc:503] Master 127.0.97.62:45367 was elected leader, sending a full tablet report...
I20260929 15:52:17.694657 8494 ts_tablet_manager.cc:1428] T 14818eb652a8457a9bfff03870784ee6 P 3a94f1c6bf464b549de59a4a1aecd307: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260929 15:52:17.696300 8493 ts_tablet_manager.cc:1428] T 14818eb652a8457a9bfff03870784ee6 P 8bc1e5f5f2da4191ab74dbdb9e29af96: Time spent starting tablet: real 0.015s user 0.012s sys 0.005s
I20260929 15:52:17.727841 8500 raft_consensus.cc:482] T 14818eb652a8457a9bfff03870784ee6 P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:52:17.728132 8500 raft_consensus.cc:504] T 14818eb652a8457a9bfff03870784ee6 P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } }
I20260929 15:52:17.729789 8500 leader_election.cc:288] T 14818eb652a8457a9bfff03870784ee6 P 8bc1e5f5f2da4191ab74dbdb9e29af96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a94f1c6bf464b549de59a4a1aecd307 (127.0.97.3:34959), b76aeb70d77b4a39b3243601bca4ffcb (127.0.97.2:38887)
I20260929 15:52:17.736533 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:17.730126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56591 (local address 127.0.97.3:34959)
0929 15:52:17.730392 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:17.730398 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:17.730694 (+ 296us) server_negotiation.cc:407] Connection header received
0929 15:52:17.730933 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:17.730942 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:17.731066 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:17.731160 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:52:17.731802 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.732703 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.734651 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.735710 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.735800 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:17.735884 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:17.736238 (+ 354us) server_negotiation.cc:299] Negotiation successful
0929 15:52:17.736326 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 15:52:17.737306 8489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:17.730856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43979 (local address 127.0.97.2:38887)
0929 15:52:17.731049 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:17.731057 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:17.731502 (+ 445us) server_negotiation.cc:407] Connection header received
0929 15:52:17.731736 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:17.731741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:17.731813 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:17.731918 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:52:17.732395 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.733486 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.735403 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.736415 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.736547 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:17.736655 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:17.736956 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 15:52:17.737079 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 15:52:17.737305 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:17.730013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34959 (local address 127.0.97.1:56591)
0929 15:52:17.730588 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:17.730609 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:17.730770 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:17.731204 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:17.731208 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:17.731235 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:52:17.731677 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:17.731693 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.732826 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.732831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:17.734494 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:17.734505 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.735898 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.735904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:17.736077 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:17.736114 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 15:52:17.736215 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 15:52:17.736365 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":288,"threads_started":1}
I20260929 15:52:17.737320 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:17.730754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38887 (local address 127.0.97.1:43979)
0929 15:52:17.731364 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:17.731395 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:17.731557 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:17.731928 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:17.731931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:17.731951 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:52:17.732239 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:17.732246 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.733610 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.733614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:17.735249 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:17.735265 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.736642 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.736649 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:17.736814 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:17.736836 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 15:52:17.736914 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 15:52:17.736992 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":1,"thread_start_us":291,"threads_started":1}
I20260929 15:52:17.738358 8279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14818eb652a8457a9bfff03870784ee6" candidate_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" is_pre_election: true
I20260929 15:52:17.738358 8423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14818eb652a8457a9bfff03870784ee6" candidate_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a94f1c6bf464b549de59a4a1aecd307" is_pre_election: true
I20260929 15:52:17.738924 8423 raft_consensus.cc:2454] T 14818eb652a8457a9bfff03870784ee6 P 3a94f1c6bf464b549de59a4a1aecd307 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bc1e5f5f2da4191ab74dbdb9e29af96 in term 0.
I20260929 15:52:17.738914 8279 raft_consensus.cc:2454] T 14818eb652a8457a9bfff03870784ee6 P b76aeb70d77b4a39b3243601bca4ffcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bc1e5f5f2da4191ab74dbdb9e29af96 in term 0.
I20260929 15:52:17.739518 8070 leader_election.cc:302] T 14818eb652a8457a9bfff03870784ee6 P 8bc1e5f5f2da4191ab74dbdb9e29af96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bc1e5f5f2da4191ab74dbdb9e29af96, b76aeb70d77b4a39b3243601bca4ffcb; no voters:
I20260929 15:52:17.739825 8500 raft_consensus.cc:2781] T 14818eb652a8457a9bfff03870784ee6 P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:52:17.739987 8500 raft_consensus.cc:482] T 14818eb652a8457a9bfff03870784ee6 P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:52:17.740091 8500 raft_consensus.cc:3037] T 14818eb652a8457a9bfff03870784ee6 P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:52:17.742300 8500 raft_consensus.cc:504] T 14818eb652a8457a9bfff03870784ee6 P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } }
I20260929 15:52:17.742846 8500 leader_election.cc:288] T 14818eb652a8457a9bfff03870784ee6 P 8bc1e5f5f2da4191ab74dbdb9e29af96 [CANDIDATE]: Term 1 election: Requested vote from peers 3a94f1c6bf464b549de59a4a1aecd307 (127.0.97.3:34959), b76aeb70d77b4a39b3243601bca4ffcb (127.0.97.2:38887)
I20260929 15:52:17.743317 8423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14818eb652a8457a9bfff03870784ee6" candidate_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a94f1c6bf464b549de59a4a1aecd307"
I20260929 15:52:17.743347 8279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14818eb652a8457a9bfff03870784ee6" candidate_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b76aeb70d77b4a39b3243601bca4ffcb"
I20260929 15:52:17.743500 8423 raft_consensus.cc:3037] T 14818eb652a8457a9bfff03870784ee6 P 3a94f1c6bf464b549de59a4a1aecd307 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:52:17.743544 8279 raft_consensus.cc:3037] T 14818eb652a8457a9bfff03870784ee6 P b76aeb70d77b4a39b3243601bca4ffcb [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:52:17.745656 8423 raft_consensus.cc:2454] T 14818eb652a8457a9bfff03870784ee6 P 3a94f1c6bf464b549de59a4a1aecd307 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bc1e5f5f2da4191ab74dbdb9e29af96 in term 1.
I20260929 15:52:17.745759 8279 raft_consensus.cc:2454] T 14818eb652a8457a9bfff03870784ee6 P b76aeb70d77b4a39b3243601bca4ffcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bc1e5f5f2da4191ab74dbdb9e29af96 in term 1.
I20260929 15:52:17.746016 8070 leader_election.cc:302] T 14818eb652a8457a9bfff03870784ee6 P 8bc1e5f5f2da4191ab74dbdb9e29af96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a94f1c6bf464b549de59a4a1aecd307, 8bc1e5f5f2da4191ab74dbdb9e29af96; no voters:
I20260929 15:52:17.746313 8500 raft_consensus.cc:2781] T 14818eb652a8457a9bfff03870784ee6 P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:52:17.746745 8500 raft_consensus.cc:686] T 14818eb652a8457a9bfff03870784ee6 P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 1 LEADER]: Becoming Leader. State: Replica: 8bc1e5f5f2da4191ab74dbdb9e29af96, State: Running, Role: LEADER
I20260929 15:52:17.747185 8500 consensus_queue.cc:237] T 14818eb652a8457a9bfff03870784ee6 P 8bc1e5f5f2da4191ab74dbdb9e29af96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } }
I20260929 15:52:17.751441 7983 catalog_manager.cc:5495] T 14818eb652a8457a9bfff03870784ee6 P 8bc1e5f5f2da4191ab74dbdb9e29af96 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8bc1e5f5f2da4191ab74dbdb9e29af96 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } health_report { overall_health: HEALTHY } } }
I20260929 15:52:17.774091 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:17.769026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39154 (local address 127.0.97.1:34987)
0929 15:52:17.769223 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:17.769229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:17.769248 (+ 19us) server_negotiation.cc:407] Connection header received
0929 15:52:17.769456 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:17.769460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:17.769540 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:17.769689 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:52:17.770025 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.770716 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.772005 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.772402 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.772533 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:17.772949 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:52:17.773507 (+ 558us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:52:17.773655 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:17.773753 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 15:52:17.773827 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 15:52:17.781608 8279 raft_consensus.cc:1261] T 14818eb652a8457a9bfff03870784ee6 P b76aeb70d77b4a39b3243601bca4ffcb [term 1 FOLLOWER]: Refusing update from remote peer 8bc1e5f5f2da4191ab74dbdb9e29af96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:52:17.781607 8423 raft_consensus.cc:1261] T 14818eb652a8457a9bfff03870784ee6 P 3a94f1c6bf464b549de59a4a1aecd307 [term 1 FOLLOWER]: Refusing update from remote peer 8bc1e5f5f2da4191ab74dbdb9e29af96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:52:17.782538 8503 consensus_queue.cc:1035] T 14818eb652a8457a9bfff03870784ee6 P 8bc1e5f5f2da4191ab74dbdb9e29af96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:52:17.782965 8500 consensus_queue.cc:1035] T 14818eb652a8457a9bfff03870784ee6 P 8bc1e5f5f2da4191ab74dbdb9e29af96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:52:17.794572 8506 mvcc.cc:205] Tried to move back new op lower bound from 7334695476344160256 to 7334695476215705600. Current Snapshot: MvccSnapshot[applied={T|T < 7334695476344160256}]
I20260929 15:52:17.812727 7983 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:52:17.812990 7983 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:52:17.816111 7983 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 035d2b0927e444dc988dd4a5b7e816bd: Sending DeleteTablet for 3 replicas of tablet 14818eb652a8457a9bfff03870784ee6
I20260929 15:52:17.817193 8398 tablet_service.cc:1514] Processing DeleteTablet for tablet 14818eb652a8457a9bfff03870784ee6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:52:17 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49876
I20260929 15:52:17.817385 8259 tablet_service.cc:1514] Processing DeleteTablet for tablet 14818eb652a8457a9bfff03870784ee6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:52:17 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48030
I20260929 15:52:17.817270 8120 tablet_service.cc:1514] Processing DeleteTablet for tablet 14818eb652a8457a9bfff03870784ee6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:52:17 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39152
I20260929 15:52:17.820232 7983 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:52:17.821014 7983 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:52:17.829761 8258 tablet_service.cc:1467] Processing CreateTablet for tablet d43a0825b39449db97017c7427f75e01 (DEFAULT_TABLE table=test-table [id=c76db3435c26475dabe7db5b9e39a93f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:52:17.829880 8397 tablet_service.cc:1467] Processing CreateTablet for tablet d43a0825b39449db97017c7427f75e01 (DEFAULT_TABLE table=test-table [id=c76db3435c26475dabe7db5b9e39a93f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:52:17.830410 8258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d43a0825b39449db97017c7427f75e01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:52:17.830425 8397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d43a0825b39449db97017c7427f75e01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:52:17.830639 8517 tablet_replica.cc:330] T 14818eb652a8457a9bfff03870784ee6 P b76aeb70d77b4a39b3243601bca4ffcb: stopping tablet replica
I20260929 15:52:17.831245 8517 raft_consensus.cc:2229] T 14818eb652a8457a9bfff03870784ee6 P b76aeb70d77b4a39b3243601bca4ffcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:52:17.831794 8517 raft_consensus.cc:2258] T 14818eb652a8457a9bfff03870784ee6 P b76aeb70d77b4a39b3243601bca4ffcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:52:17.833532 8518 tablet_replica.cc:330] T 14818eb652a8457a9bfff03870784ee6 P 8bc1e5f5f2da4191ab74dbdb9e29af96: stopping tablet replica
I20260929 15:52:17.833925 8518 raft_consensus.cc:2229] T 14818eb652a8457a9bfff03870784ee6 P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:52:17.833873 8119 tablet_service.cc:1467] Processing CreateTablet for tablet d43a0825b39449db97017c7427f75e01 (DEFAULT_TABLE table=test-table [id=c76db3435c26475dabe7db5b9e39a93f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:52:17.834472 8119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d43a0825b39449db97017c7427f75e01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:52:17.834597 8518 raft_consensus.cc:2258] T 14818eb652a8457a9bfff03870784ee6 P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:52:17.835323 8517 ts_tablet_manager.cc:1905] T 14818eb652a8457a9bfff03870784ee6 P b76aeb70d77b4a39b3243601bca4ffcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:52:17.837366 8492 tablet_bootstrap.cc:492] T d43a0825b39449db97017c7427f75e01 P b76aeb70d77b4a39b3243601bca4ffcb: Bootstrap starting.
I20260929 15:52:17.839531 8492 tablet_bootstrap.cc:654] T d43a0825b39449db97017c7427f75e01 P b76aeb70d77b4a39b3243601bca4ffcb: Neither blocks nor log segments found. Creating new log.
I20260929 15:52:17.842685 8517 ts_tablet_manager.cc:1918] T 14818eb652a8457a9bfff03870784ee6 P b76aeb70d77b4a39b3243601bca4ffcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:52:17.842846 8517 log.cc:1196] T 14818eb652a8457a9bfff03870784ee6 P b76aeb70d77b4a39b3243601bca4ffcb: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-1/wal/wals/14818eb652a8457a9bfff03870784ee6
I20260929 15:52:17.843050 8518 ts_tablet_manager.cc:1905] T 14818eb652a8457a9bfff03870784ee6 P 8bc1e5f5f2da4191ab74dbdb9e29af96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:52:17.843377 8517 ts_tablet_manager.cc:1939] T 14818eb652a8457a9bfff03870784ee6 P b76aeb70d77b4a39b3243601bca4ffcb: Deleting consensus metadata
I20260929 15:52:17.844914 8493 tablet_bootstrap.cc:492] T d43a0825b39449db97017c7427f75e01 P 8bc1e5f5f2da4191ab74dbdb9e29af96: Bootstrap starting.
I20260929 15:52:17.845968 8494 tablet_bootstrap.cc:492] T d43a0825b39449db97017c7427f75e01 P 3a94f1c6bf464b549de59a4a1aecd307: Bootstrap starting.
I20260929 15:52:17.846527 8493 tablet_bootstrap.cc:654] T d43a0825b39449db97017c7427f75e01 P 8bc1e5f5f2da4191ab74dbdb9e29af96: Neither blocks nor log segments found. Creating new log.
I20260929 15:52:17.847740 8494 tablet_bootstrap.cc:654] T d43a0825b39449db97017c7427f75e01 P 3a94f1c6bf464b549de59a4a1aecd307: Neither blocks nor log segments found. Creating new log.
I20260929 15:52:17.851401 7967 catalog_manager.cc:4841] TS b76aeb70d77b4a39b3243601bca4ffcb (127.0.97.2:38887): tablet 14818eb652a8457a9bfff03870784ee6 (table test-table [id=370adf68136c441aa27e9eeaccbdc432]) successfully deleted
I20260929 15:52:17.855808 8492 tablet_bootstrap.cc:492] T d43a0825b39449db97017c7427f75e01 P b76aeb70d77b4a39b3243601bca4ffcb: No bootstrap required, opened a new log
I20260929 15:52:17.856060 8492 ts_tablet_manager.cc:1397] T d43a0825b39449db97017c7427f75e01 P b76aeb70d77b4a39b3243601bca4ffcb: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.015s
I20260929 15:52:17.856353 8494 tablet_bootstrap.cc:492] T d43a0825b39449db97017c7427f75e01 P 3a94f1c6bf464b549de59a4a1aecd307: No bootstrap required, opened a new log
I20260929 15:52:17.856498 8494 ts_tablet_manager.cc:1397] T d43a0825b39449db97017c7427f75e01 P 3a94f1c6bf464b549de59a4a1aecd307: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260929 15:52:17.857103 8492 raft_consensus.cc:348] T d43a0825b39449db97017c7427f75e01 P b76aeb70d77b4a39b3243601bca4ffcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } }
I20260929 15:52:17.857309 8494 raft_consensus.cc:348] T d43a0825b39449db97017c7427f75e01 P 3a94f1c6bf464b549de59a4a1aecd307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } }
I20260929 15:52:17.857568 8494 raft_consensus.cc:374] T d43a0825b39449db97017c7427f75e01 P 3a94f1c6bf464b549de59a4a1aecd307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:52:17.857506 8492 raft_consensus.cc:374] T d43a0825b39449db97017c7427f75e01 P b76aeb70d77b4a39b3243601bca4ffcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:52:17.857668 8494 raft_consensus.cc:729] T d43a0825b39449db97017c7427f75e01 P 3a94f1c6bf464b549de59a4a1aecd307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a94f1c6bf464b549de59a4a1aecd307, State: Initialized, Role: FOLLOWER
I20260929 15:52:17.857717 8492 raft_consensus.cc:729] T d43a0825b39449db97017c7427f75e01 P b76aeb70d77b4a39b3243601bca4ffcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b76aeb70d77b4a39b3243601bca4ffcb, State: Initialized, Role: FOLLOWER
I20260929 15:52:17.857990 8494 consensus_queue.cc:260] T d43a0825b39449db97017c7427f75e01 P 3a94f1c6bf464b549de59a4a1aecd307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } }
I20260929 15:52:17.858073 8492 consensus_queue.cc:260] T d43a0825b39449db97017c7427f75e01 P b76aeb70d77b4a39b3243601bca4ffcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } }
I20260929 15:52:17.859966 8492 ts_tablet_manager.cc:1428] T d43a0825b39449db97017c7427f75e01 P b76aeb70d77b4a39b3243601bca4ffcb: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260929 15:52:17.860841 8518 ts_tablet_manager.cc:1918] T 14818eb652a8457a9bfff03870784ee6 P 8bc1e5f5f2da4191ab74dbdb9e29af96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:52:17.861014 8518 log.cc:1196] T 14818eb652a8457a9bfff03870784ee6 P 8bc1e5f5f2da4191ab74dbdb9e29af96: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-0/wal/wals/14818eb652a8457a9bfff03870784ee6
I20260929 15:52:17.861104 8494 ts_tablet_manager.cc:1428] T d43a0825b39449db97017c7427f75e01 P 3a94f1c6bf464b549de59a4a1aecd307: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260929 15:52:17.861579 8518 ts_tablet_manager.cc:1939] T 14818eb652a8457a9bfff03870784ee6 P 8bc1e5f5f2da4191ab74dbdb9e29af96: Deleting consensus metadata
I20260929 15:52:17.861826 8516 tablet_replica.cc:330] T 14818eb652a8457a9bfff03870784ee6 P 3a94f1c6bf464b549de59a4a1aecd307: stopping tablet replica
I20260929 15:52:17.862265 8516 raft_consensus.cc:2229] T 14818eb652a8457a9bfff03870784ee6 P 3a94f1c6bf464b549de59a4a1aecd307 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:52:17.862802 8516 raft_consensus.cc:2258] T 14818eb652a8457a9bfff03870784ee6 P 3a94f1c6bf464b549de59a4a1aecd307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:52:17.863971 7967 catalog_manager.cc:4841] TS 8bc1e5f5f2da4191ab74dbdb9e29af96 (127.0.97.1:34987): tablet 14818eb652a8457a9bfff03870784ee6 (table test-table [id=370adf68136c441aa27e9eeaccbdc432]) successfully deleted
I20260929 15:52:17.866771 8493 tablet_bootstrap.cc:492] T d43a0825b39449db97017c7427f75e01 P 8bc1e5f5f2da4191ab74dbdb9e29af96: No bootstrap required, opened a new log
I20260929 15:52:17.866938 8493 ts_tablet_manager.cc:1397] T d43a0825b39449db97017c7427f75e01 P 8bc1e5f5f2da4191ab74dbdb9e29af96: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.000s
I20260929 15:52:17.867769 8493 raft_consensus.cc:348] T d43a0825b39449db97017c7427f75e01 P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } }
I20260929 15:52:17.868007 8493 raft_consensus.cc:374] T d43a0825b39449db97017c7427f75e01 P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:52:17.868105 8493 raft_consensus.cc:729] T d43a0825b39449db97017c7427f75e01 P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bc1e5f5f2da4191ab74dbdb9e29af96, State: Initialized, Role: FOLLOWER
I20260929 15:52:17.868140 8516 ts_tablet_manager.cc:1905] T 14818eb652a8457a9bfff03870784ee6 P 3a94f1c6bf464b549de59a4a1aecd307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:52:17.868394 8493 consensus_queue.cc:260] T d43a0825b39449db97017c7427f75e01 P 8bc1e5f5f2da4191ab74dbdb9e29af96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } }
I20260929 15:52:17.869159 8493 ts_tablet_manager.cc:1428] T d43a0825b39449db97017c7427f75e01 P 8bc1e5f5f2da4191ab74dbdb9e29af96: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 15:52:17.874368 8516 ts_tablet_manager.cc:1918] T 14818eb652a8457a9bfff03870784ee6 P 3a94f1c6bf464b549de59a4a1aecd307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:52:17.874548 8516 log.cc:1196] T 14818eb652a8457a9bfff03870784ee6 P 3a94f1c6bf464b549de59a4a1aecd307: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-2/wal/wals/14818eb652a8457a9bfff03870784ee6
I20260929 15:52:17.874977 8516 ts_tablet_manager.cc:1939] T 14818eb652a8457a9bfff03870784ee6 P 3a94f1c6bf464b549de59a4a1aecd307: Deleting consensus metadata
I20260929 15:52:17.876374 7967 catalog_manager.cc:4841] TS 3a94f1c6bf464b549de59a4a1aecd307 (127.0.97.3:34959): tablet 14818eb652a8457a9bfff03870784ee6 (table test-table [id=370adf68136c441aa27e9eeaccbdc432]) successfully deleted
I20260929 15:52:17.992712 8498 raft_consensus.cc:482] T d43a0825b39449db97017c7427f75e01 P b76aeb70d77b4a39b3243601bca4ffcb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:52:17.992950 8498 raft_consensus.cc:504] T d43a0825b39449db97017c7427f75e01 P b76aeb70d77b4a39b3243601bca4ffcb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } }
I20260929 15:52:17.994426 8498 leader_election.cc:288] T d43a0825b39449db97017c7427f75e01 P b76aeb70d77b4a39b3243601bca4ffcb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8bc1e5f5f2da4191ab74dbdb9e29af96 (127.0.97.1:34987), 3a94f1c6bf464b549de59a4a1aecd307 (127.0.97.3:34959)
I20260929 15:52:18.002990 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:17.995336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34959 (local address 127.0.97.2:39689)
0929 15:52:17.996061 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:17.996077 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:17.996196 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:17.996560 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:17.996567 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:17.996596 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:52:17.996930 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:17.996938 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.997968 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.997976 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:18.000523 (+ 2547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:18.000536 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:18.002118 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:18.002124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:18.002311 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:18.002347 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 15:52:18.002448 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 15:52:18.002587 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"mutex_wait_us":139,"thread_start_us":423,"threads_started":1}
I20260929 15:52:18.003000 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:17.995455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39689 (local address 127.0.97.3:34959)
0929 15:52:17.995629 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:17.995634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:17.996164 (+ 530us) server_negotiation.cc:407] Connection header received
0929 15:52:17.996343 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:17.996348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:17.996429 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:17.996525 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:52:17.997083 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.997846 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:18.000688 (+ 2842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:18.001908 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:18.002043 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:18.002147 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:18.002572 (+ 425us) server_negotiation.cc:299] Negotiation successful
0929 15:52:18.002719 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 15:52:18.003961 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:17.994989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37631 (local address 127.0.97.1:34987)
0929 15:52:17.995192 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:17.995199 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:17.995356 (+ 157us) server_negotiation.cc:407] Connection header received
0929 15:52:17.995647 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:17.995654 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:17.995748 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:17.995823 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:52:17.996703 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.997895 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:18.001462 (+ 3567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:18.002728 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:18.002862 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:18.002960 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:18.003493 (+ 533us) server_negotiation.cc:299] Negotiation successful
0929 15:52:18.003619 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 15:52:18.003991 8522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:17.994704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34987 (local address 127.0.97.2:37631)
0929 15:52:17.995255 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:17.995275 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:17.995475 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:17.996063 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:17.996068 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:17.996100 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:52:17.996478 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:17.996509 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:17.998038 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:17.998043 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:18.000522 (+ 2479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:18.000536 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:18.003117 (+ 2581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:18.003125 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:18.003292 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:18.003322 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 15:52:18.003464 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 15:52:18.003582 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":110,"thread_start_us":247,"threads_started":1}
I20260929 15:52:18.003981 8423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43a0825b39449db97017c7427f75e01" candidate_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a94f1c6bf464b549de59a4a1aecd307" is_pre_election: true
I20260929 15:52:18.004227 8423 raft_consensus.cc:2454] T d43a0825b39449db97017c7427f75e01 P 3a94f1c6bf464b549de59a4a1aecd307 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b76aeb70d77b4a39b3243601bca4ffcb in term 0.
I20260929 15:52:18.004695 8210 leader_election.cc:302] T d43a0825b39449db97017c7427f75e01 P b76aeb70d77b4a39b3243601bca4ffcb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a94f1c6bf464b549de59a4a1aecd307, b76aeb70d77b4a39b3243601bca4ffcb; no voters:
I20260929 15:52:18.005098 8498 raft_consensus.cc:2781] T d43a0825b39449db97017c7427f75e01 P b76aeb70d77b4a39b3243601bca4ffcb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:52:18.005219 8498 raft_consensus.cc:482] T d43a0825b39449db97017c7427f75e01 P b76aeb70d77b4a39b3243601bca4ffcb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:52:18.005322 8498 raft_consensus.cc:3037] T d43a0825b39449db97017c7427f75e01 P b76aeb70d77b4a39b3243601bca4ffcb [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:52:18.005185 8145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43a0825b39449db97017c7427f75e01" candidate_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" is_pre_election: true
I20260929 15:52:18.005666 8145 raft_consensus.cc:2454] T d43a0825b39449db97017c7427f75e01 P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b76aeb70d77b4a39b3243601bca4ffcb in term 0.
I20260929 15:52:18.008055 8498 raft_consensus.cc:504] T d43a0825b39449db97017c7427f75e01 P b76aeb70d77b4a39b3243601bca4ffcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } }
I20260929 15:52:18.008639 8498 leader_election.cc:288] T d43a0825b39449db97017c7427f75e01 P b76aeb70d77b4a39b3243601bca4ffcb [CANDIDATE]: Term 1 election: Requested vote from peers 8bc1e5f5f2da4191ab74dbdb9e29af96 (127.0.97.1:34987), 3a94f1c6bf464b549de59a4a1aecd307 (127.0.97.3:34959)
I20260929 15:52:18.009193 8423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43a0825b39449db97017c7427f75e01" candidate_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a94f1c6bf464b549de59a4a1aecd307"
I20260929 15:52:18.009266 8145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43a0825b39449db97017c7427f75e01" candidate_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96"
I20260929 15:52:18.009402 8423 raft_consensus.cc:3037] T d43a0825b39449db97017c7427f75e01 P 3a94f1c6bf464b549de59a4a1aecd307 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:52:18.009501 8145 raft_consensus.cc:3037] T d43a0825b39449db97017c7427f75e01 P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:52:18.011543 8423 raft_consensus.cc:2454] T d43a0825b39449db97017c7427f75e01 P 3a94f1c6bf464b549de59a4a1aecd307 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b76aeb70d77b4a39b3243601bca4ffcb in term 1.
I20260929 15:52:18.011983 8210 leader_election.cc:302] T d43a0825b39449db97017c7427f75e01 P b76aeb70d77b4a39b3243601bca4ffcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a94f1c6bf464b549de59a4a1aecd307, b76aeb70d77b4a39b3243601bca4ffcb; no voters:
I20260929 15:52:18.012249 8498 raft_consensus.cc:2781] T d43a0825b39449db97017c7427f75e01 P b76aeb70d77b4a39b3243601bca4ffcb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:52:18.012792 8498 raft_consensus.cc:686] T d43a0825b39449db97017c7427f75e01 P b76aeb70d77b4a39b3243601bca4ffcb [term 1 LEADER]: Becoming Leader. State: Replica: b76aeb70d77b4a39b3243601bca4ffcb, State: Running, Role: LEADER
I20260929 15:52:18.012902 8145 raft_consensus.cc:2454] T d43a0825b39449db97017c7427f75e01 P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b76aeb70d77b4a39b3243601bca4ffcb in term 1.
I20260929 15:52:18.013542 8498 consensus_queue.cc:237] T d43a0825b39449db97017c7427f75e01 P b76aeb70d77b4a39b3243601bca4ffcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } }
I20260929 15:52:18.017328 7983 catalog_manager.cc:5495] T d43a0825b39449db97017c7427f75e01 P b76aeb70d77b4a39b3243601bca4ffcb reported cstate change: term changed from 0 to 1, leader changed from <none> to b76aeb70d77b4a39b3243601bca4ffcb (127.0.97.2). New cstate: current_term: 1 leader_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } health_report { overall_health: UNKNOWN } } }
W20260929 15:52:18.050565 8469 tablet.cc:2369] T d43a0825b39449db97017c7427f75e01 P 3a94f1c6bf464b549de59a4a1aecd307: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 15:52:18 dist-test-slave-r0ck krb5kdc[7938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697135, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
W20260929 15:52:18.061089 8330 tablet.cc:2369] T d43a0825b39449db97017c7427f75e01 P b76aeb70d77b4a39b3243601bca4ffcb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:52:18.063495 8191 tablet.cc:2369] T d43a0825b39449db97017c7427f75e01 P 8bc1e5f5f2da4191ab74dbdb9e29af96: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:52:18.064802 8531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:18.047652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45367 (local address 127.0.0.1:60804)
0929 15:52:18.048328 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:18.048345 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:18.048468 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:18.048835 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:18.048838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:18.049550 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:18.049802 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:18.049811 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:18.051235 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:18.051243 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:18.052713 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:18.052726 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:18.053705 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:18.053712 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:18.053874 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:18.054668 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:18.054690 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:18.057822 (+ 3132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:18.061822 (+ 4000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:18.061835 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:18.061843 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:18.062431 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:18.062818 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:18.062823 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:18.062826 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:18.063001 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:18.063710 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:18.063718 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:18.063880 (+ 162us) client_negotiation.cc:769] Sending connection context
0929 15:52:18.064053 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 15:52:18.064341 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":372,"threads_started":1}
I20260929 15:52:18.064888 8077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:18.047655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60804 (local address 127.0.97.62:45367)
0929 15:52:18.047886 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:18.047892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:18.048437 (+ 545us) server_negotiation.cc:407] Connection header received
0929 15:52:18.048583 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:18.048590 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:18.048672 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:18.048794 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:18.049969 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:18.051088 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:18.052869 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:18.053494 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:18.053624 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:18.058015 (+ 4391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:18.058022 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:18.058027 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:18.058062 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:18.061678 (+ 3616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:18.062558 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:18.062564 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:18.062568 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:18.062695 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:18.063109 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:18.063114 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:18.063116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:18.063445 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:18.063580 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:18.064200 (+ 620us) server_negotiation.cc:299] Negotiation successful
0929 15:52:18.064501 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":106}
I20260929 15:52:18.070861 7983 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60804:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 15:52:18.072000 7983 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:52:18.079305 8119 tablet_service.cc:1467] Processing CreateTablet for tablet 35cc27f784a54b12b613090c602beea0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=60dc106656af455e9a7a1a0db7fc23ef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 15:52:18.079396 8259 tablet_service.cc:1467] Processing CreateTablet for tablet 35cc27f784a54b12b613090c602beea0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=60dc106656af455e9a7a1a0db7fc23ef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 15:52:18.079831 8119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35cc27f784a54b12b613090c602beea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:52:18.079742 8398 tablet_service.cc:1467] Processing CreateTablet for tablet 35cc27f784a54b12b613090c602beea0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=60dc106656af455e9a7a1a0db7fc23ef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 15:52:18.079912 8259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35cc27f784a54b12b613090c602beea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:52:18.080214 8398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35cc27f784a54b12b613090c602beea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:52:18.086354 8492 tablet_bootstrap.cc:492] T 35cc27f784a54b12b613090c602beea0 P b76aeb70d77b4a39b3243601bca4ffcb: Bootstrap starting.
I20260929 15:52:18.087548 8494 tablet_bootstrap.cc:492] T 35cc27f784a54b12b613090c602beea0 P 3a94f1c6bf464b549de59a4a1aecd307: Bootstrap starting.
I20260929 15:52:18.087988 8492 tablet_bootstrap.cc:654] T 35cc27f784a54b12b613090c602beea0 P b76aeb70d77b4a39b3243601bca4ffcb: Neither blocks nor log segments found. Creating new log.
I20260929 15:52:18.088969 8493 tablet_bootstrap.cc:492] T 35cc27f784a54b12b613090c602beea0 P 8bc1e5f5f2da4191ab74dbdb9e29af96: Bootstrap starting.
I20260929 15:52:18.089126 8494 tablet_bootstrap.cc:654] T 35cc27f784a54b12b613090c602beea0 P 3a94f1c6bf464b549de59a4a1aecd307: Neither blocks nor log segments found. Creating new log.
I20260929 15:52:18.090956 8493 tablet_bootstrap.cc:654] T 35cc27f784a54b12b613090c602beea0 P 8bc1e5f5f2da4191ab74dbdb9e29af96: Neither blocks nor log segments found. Creating new log.
I20260929 15:52:18.099747 8492 tablet_bootstrap.cc:492] T 35cc27f784a54b12b613090c602beea0 P b76aeb70d77b4a39b3243601bca4ffcb: No bootstrap required, opened a new log
I20260929 15:52:18.099931 8492 ts_tablet_manager.cc:1397] T 35cc27f784a54b12b613090c602beea0 P b76aeb70d77b4a39b3243601bca4ffcb: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260929 15:52:18.100637 8492 raft_consensus.cc:348] T 35cc27f784a54b12b613090c602beea0 P b76aeb70d77b4a39b3243601bca4ffcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } }
I20260929 15:52:18.100893 8492 raft_consensus.cc:374] T 35cc27f784a54b12b613090c602beea0 P b76aeb70d77b4a39b3243601bca4ffcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:52:18.100972 8492 raft_consensus.cc:729] T 35cc27f784a54b12b613090c602beea0 P b76aeb70d77b4a39b3243601bca4ffcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b76aeb70d77b4a39b3243601bca4ffcb, State: Initialized, Role: FOLLOWER
I20260929 15:52:18.101248 8492 consensus_queue.cc:260] T 35cc27f784a54b12b613090c602beea0 P b76aeb70d77b4a39b3243601bca4ffcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } }
I20260929 15:52:18.101679 8498 tablet_replica.cc:439] T 35cc27f784a54b12b613090c602beea0 P b76aeb70d77b4a39b3243601bca4ffcb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } } }
I20260929 15:52:18.101953 8498 tablet_replica.cc:442] T 35cc27f784a54b12b613090c602beea0 P b76aeb70d77b4a39b3243601bca4ffcb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:52:18.102581 8493 tablet_bootstrap.cc:492] T 35cc27f784a54b12b613090c602beea0 P 8bc1e5f5f2da4191ab74dbdb9e29af96: No bootstrap required, opened a new log
I20260929 15:52:18.102782 8493 ts_tablet_manager.cc:1397] T 35cc27f784a54b12b613090c602beea0 P 8bc1e5f5f2da4191ab74dbdb9e29af96: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20260929 15:52:18.103829 8493 raft_consensus.cc:348] T 35cc27f784a54b12b613090c602beea0 P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } }
I20260929 15:52:18.104197 8493 raft_consensus.cc:374] T 35cc27f784a54b12b613090c602beea0 P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:52:18.104312 8493 raft_consensus.cc:729] T 35cc27f784a54b12b613090c602beea0 P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bc1e5f5f2da4191ab74dbdb9e29af96, State: Initialized, Role: FOLLOWER
I20260929 15:52:18.104562 8492 ts_tablet_manager.cc:1428] T 35cc27f784a54b12b613090c602beea0 P b76aeb70d77b4a39b3243601bca4ffcb: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260929 15:52:18.105490 8493 consensus_queue.cc:260] T 35cc27f784a54b12b613090c602beea0 P 8bc1e5f5f2da4191ab74dbdb9e29af96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } }
I20260929 15:52:18.106030 8503 tablet_replica.cc:439] T 35cc27f784a54b12b613090c602beea0 P 8bc1e5f5f2da4191ab74dbdb9e29af96: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } } }
I20260929 15:52:18.106441 8503 tablet_replica.cc:442] T 35cc27f784a54b12b613090c602beea0 P 8bc1e5f5f2da4191ab74dbdb9e29af96: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:52:18.107699 8493 ts_tablet_manager.cc:1428] T 35cc27f784a54b12b613090c602beea0 P 8bc1e5f5f2da4191ab74dbdb9e29af96: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260929 15:52:18.109601 8494 tablet_bootstrap.cc:492] T 35cc27f784a54b12b613090c602beea0 P 3a94f1c6bf464b549de59a4a1aecd307: No bootstrap required, opened a new log
I20260929 15:52:18.109793 8494 ts_tablet_manager.cc:1397] T 35cc27f784a54b12b613090c602beea0 P 3a94f1c6bf464b549de59a4a1aecd307: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.016s
I20260929 15:52:18.110411 8494 raft_consensus.cc:348] T 35cc27f784a54b12b613090c602beea0 P 3a94f1c6bf464b549de59a4a1aecd307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } }
I20260929 15:52:18.110646 8494 raft_consensus.cc:374] T 35cc27f784a54b12b613090c602beea0 P 3a94f1c6bf464b549de59a4a1aecd307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:52:18.110733 8494 raft_consensus.cc:729] T 35cc27f784a54b12b613090c602beea0 P 3a94f1c6bf464b549de59a4a1aecd307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a94f1c6bf464b549de59a4a1aecd307, State: Initialized, Role: FOLLOWER
I20260929 15:52:18.110971 8494 consensus_queue.cc:260] T 35cc27f784a54b12b613090c602beea0 P 3a94f1c6bf464b549de59a4a1aecd307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } }
I20260929 15:52:18.111502 8494 ts_tablet_manager.cc:1428] T 35cc27f784a54b12b613090c602beea0 P 3a94f1c6bf464b549de59a4a1aecd307: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 15:52:18.111382 8499 tablet_replica.cc:439] T 35cc27f784a54b12b613090c602beea0 P 3a94f1c6bf464b549de59a4a1aecd307: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } } }
I20260929 15:52:18.111694 8499 tablet_replica.cc:442] T 35cc27f784a54b12b613090c602beea0 P 3a94f1c6bf464b549de59a4a1aecd307: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:52:18.207844 8498 raft_consensus.cc:482] T 35cc27f784a54b12b613090c602beea0 P b76aeb70d77b4a39b3243601bca4ffcb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:52:18.208057 8498 raft_consensus.cc:504] T 35cc27f784a54b12b613090c602beea0 P b76aeb70d77b4a39b3243601bca4ffcb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } }
I20260929 15:52:18.208789 8498 leader_election.cc:288] T 35cc27f784a54b12b613090c602beea0 P b76aeb70d77b4a39b3243601bca4ffcb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8bc1e5f5f2da4191ab74dbdb9e29af96 (127.0.97.1:34987), 3a94f1c6bf464b549de59a4a1aecd307 (127.0.97.3:34959)
I20260929 15:52:18.209304 8145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cc27f784a54b12b613090c602beea0" candidate_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" is_pre_election: true
I20260929 15:52:18.209431 8423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cc27f784a54b12b613090c602beea0" candidate_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a94f1c6bf464b549de59a4a1aecd307" is_pre_election: true
I20260929 15:52:18.209602 8145 raft_consensus.cc:2454] T 35cc27f784a54b12b613090c602beea0 P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b76aeb70d77b4a39b3243601bca4ffcb in term 0.
I20260929 15:52:18.209704 8423 raft_consensus.cc:2454] T 35cc27f784a54b12b613090c602beea0 P 3a94f1c6bf464b549de59a4a1aecd307 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b76aeb70d77b4a39b3243601bca4ffcb in term 0.
I20260929 15:52:18.210071 8210 leader_election.cc:302] T 35cc27f784a54b12b613090c602beea0 P b76aeb70d77b4a39b3243601bca4ffcb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bc1e5f5f2da4191ab74dbdb9e29af96, b76aeb70d77b4a39b3243601bca4ffcb; no voters:
I20260929 15:52:18.210346 8498 raft_consensus.cc:2781] T 35cc27f784a54b12b613090c602beea0 P b76aeb70d77b4a39b3243601bca4ffcb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:52:18.210444 8498 raft_consensus.cc:482] T 35cc27f784a54b12b613090c602beea0 P b76aeb70d77b4a39b3243601bca4ffcb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:52:18.210597 8498 raft_consensus.cc:3037] T 35cc27f784a54b12b613090c602beea0 P b76aeb70d77b4a39b3243601bca4ffcb [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:52:18.212527 8498 raft_consensus.cc:504] T 35cc27f784a54b12b613090c602beea0 P b76aeb70d77b4a39b3243601bca4ffcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } }
I20260929 15:52:18.213095 8498 leader_election.cc:288] T 35cc27f784a54b12b613090c602beea0 P b76aeb70d77b4a39b3243601bca4ffcb [CANDIDATE]: Term 1 election: Requested vote from peers 8bc1e5f5f2da4191ab74dbdb9e29af96 (127.0.97.1:34987), 3a94f1c6bf464b549de59a4a1aecd307 (127.0.97.3:34959)
I20260929 15:52:18.213527 8145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cc27f784a54b12b613090c602beea0" candidate_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96"
I20260929 15:52:18.213637 8423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cc27f784a54b12b613090c602beea0" candidate_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a94f1c6bf464b549de59a4a1aecd307"
I20260929 15:52:18.213749 8145 raft_consensus.cc:3037] T 35cc27f784a54b12b613090c602beea0 P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:52:18.213866 8423 raft_consensus.cc:3037] T 35cc27f784a54b12b613090c602beea0 P 3a94f1c6bf464b549de59a4a1aecd307 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:52:18.216212 8145 raft_consensus.cc:2454] T 35cc27f784a54b12b613090c602beea0 P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b76aeb70d77b4a39b3243601bca4ffcb in term 1.
I20260929 15:52:18.216586 8210 leader_election.cc:302] T 35cc27f784a54b12b613090c602beea0 P b76aeb70d77b4a39b3243601bca4ffcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bc1e5f5f2da4191ab74dbdb9e29af96, b76aeb70d77b4a39b3243601bca4ffcb; no voters:
I20260929 15:52:18.216766 8423 raft_consensus.cc:2454] T 35cc27f784a54b12b613090c602beea0 P 3a94f1c6bf464b549de59a4a1aecd307 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b76aeb70d77b4a39b3243601bca4ffcb in term 1.
I20260929 15:52:18.216838 8498 raft_consensus.cc:2781] T 35cc27f784a54b12b613090c602beea0 P b76aeb70d77b4a39b3243601bca4ffcb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:52:18.217422 8498 raft_consensus.cc:686] T 35cc27f784a54b12b613090c602beea0 P b76aeb70d77b4a39b3243601bca4ffcb [term 1 LEADER]: Becoming Leader. State: Replica: b76aeb70d77b4a39b3243601bca4ffcb, State: Running, Role: LEADER
I20260929 15:52:18.217669 8498 consensus_queue.cc:237] T 35cc27f784a54b12b613090c602beea0 P b76aeb70d77b4a39b3243601bca4ffcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } }
I20260929 15:52:18.219354 8524 tablet_replica.cc:439] T 35cc27f784a54b12b613090c602beea0 P b76aeb70d77b4a39b3243601bca4ffcb: TxnStatusTablet state changed. Reason: New leader b76aeb70d77b4a39b3243601bca4ffcb. Latest consensus state: current_term: 1 leader_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } } }
I20260929 15:52:18.219759 8524 tablet_replica.cc:442] T 35cc27f784a54b12b613090c602beea0 P b76aeb70d77b4a39b3243601bca4ffcb: This TxnStatusTablet replica's current role is: LEADER
I20260929 15:52:18.220616 7983 catalog_manager.cc:5495] T 35cc27f784a54b12b613090c602beea0 P b76aeb70d77b4a39b3243601bca4ffcb reported cstate change: term changed from 0 to 1, leader changed from <none> to b76aeb70d77b4a39b3243601bca4ffcb (127.0.97.2). New cstate: current_term: 1 leader_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } health_report { overall_health: UNKNOWN } } }
I20260929 15:52:18.221803 8536 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 15:52:18.222028 8536 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 15:52:18.260591 8531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:18.246944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38887 (local address 127.0.0.1:48038)
0929 15:52:18.247163 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:18.247185 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:18.247326 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:18.254244 (+ 6918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:18.254250 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:18.254265 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:52:18.254468 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:18.254480 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:18.256040 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:18.256047 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:18.257692 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:18.257708 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:18.258728 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:18.258734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:18.258927 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:18.258997 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:52:18.260039 (+ 1042us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:52:18.260059 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 15:52:18.260118 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 15:52:18.260275 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 15:52:18.261547 8539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:18.247262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48038 (local address 127.0.97.2:38887)
0929 15:52:18.253792 (+ 6530us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:18.253799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:18.253819 (+ 20us) server_negotiation.cc:407] Connection header received
0929 15:52:18.253916 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:18.253921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:18.254024 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:18.254190 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:52:18.254667 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:18.255893 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:18.257877 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:18.258559 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:18.258691 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:18.259203 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:52:18.259899 (+ 696us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:52:18.261001 (+ 1102us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:18.261081 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 15:52:18.261168 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6389,"thread_start_us":6249,"threads_started":1}
I20260929 15:52:18.265861 8423 raft_consensus.cc:1261] T 35cc27f784a54b12b613090c602beea0 P 3a94f1c6bf464b549de59a4a1aecd307 [term 1 FOLLOWER]: Refusing update from remote peer b76aeb70d77b4a39b3243601bca4ffcb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:52:18.266172 8145 raft_consensus.cc:1261] T 35cc27f784a54b12b613090c602beea0 P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 1 FOLLOWER]: Refusing update from remote peer b76aeb70d77b4a39b3243601bca4ffcb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:52:18.266563 8498 consensus_queue.cc:1035] T 35cc27f784a54b12b613090c602beea0 P b76aeb70d77b4a39b3243601bca4ffcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:52:18.267025 8524 consensus_queue.cc:1035] T 35cc27f784a54b12b613090c602beea0 P b76aeb70d77b4a39b3243601bca4ffcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:52:18.277812 8503 tablet_replica.cc:439] T 35cc27f784a54b12b613090c602beea0 P 8bc1e5f5f2da4191ab74dbdb9e29af96: TxnStatusTablet state changed. Reason: New leader b76aeb70d77b4a39b3243601bca4ffcb. Latest consensus state: current_term: 1 leader_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } } }
I20260929 15:52:18.278069 8503 tablet_replica.cc:442] T 35cc27f784a54b12b613090c602beea0 P 8bc1e5f5f2da4191ab74dbdb9e29af96: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:52:18.280205 8499 tablet_replica.cc:439] T 35cc27f784a54b12b613090c602beea0 P 3a94f1c6bf464b549de59a4a1aecd307: TxnStatusTablet state changed. Reason: New leader b76aeb70d77b4a39b3243601bca4ffcb. Latest consensus state: current_term: 1 leader_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } } }
I20260929 15:52:18.280296 8524 tablet_replica.cc:439] T 35cc27f784a54b12b613090c602beea0 P b76aeb70d77b4a39b3243601bca4ffcb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } } }
I20260929 15:52:18.280441 8499 tablet_replica.cc:442] T 35cc27f784a54b12b613090c602beea0 P 3a94f1c6bf464b549de59a4a1aecd307: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:52:18.280512 8524 tablet_replica.cc:442] T 35cc27f784a54b12b613090c602beea0 P b76aeb70d77b4a39b3243601bca4ffcb: This TxnStatusTablet replica's current role is: LEADER
I20260929 15:52:18.280750 8508 mvcc.cc:205] Tried to move back new op lower bound from 7334695478329339904 to 7334695478141796352. Current Snapshot: MvccSnapshot[applied={T|T < 7334695478329339904 or (T in {7334695478329339904})}]
I20260929 15:52:18.283007 8503 tablet_replica.cc:439] T 35cc27f784a54b12b613090c602beea0 P 8bc1e5f5f2da4191ab74dbdb9e29af96: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } } }
I20260929 15:52:18.283242 8503 tablet_replica.cc:442] T 35cc27f784a54b12b613090c602beea0 P 8bc1e5f5f2da4191ab74dbdb9e29af96: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:52:18.297705 8507 mvcc.cc:205] Tried to move back new op lower bound from 7334695478329339904 to 7334695478141796352. Current Snapshot: MvccSnapshot[applied={T|T < 7334695478329339904}]
I20260929 15:52:18.297996 8499 tablet_replica.cc:439] T 35cc27f784a54b12b613090c602beea0 P 3a94f1c6bf464b549de59a4a1aecd307: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } } }
I20260929 15:52:18.298193 8499 tablet_replica.cc:442] T 35cc27f784a54b12b613090c602beea0 P 3a94f1c6bf464b549de59a4a1aecd307: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:52:18.303479 8542 tablet_replica.cc:439] T 35cc27f784a54b12b613090c602beea0 P b76aeb70d77b4a39b3243601bca4ffcb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } } }
I20260929 15:52:18.304102 8542 tablet_replica.cc:442] T 35cc27f784a54b12b613090c602beea0 P b76aeb70d77b4a39b3243601bca4ffcb: This TxnStatusTablet replica's current role is: LEADER
I20260929 15:52:18.309716 8539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:18.297529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48040 (local address 127.0.97.2:38887)
0929 15:52:18.298291 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:18.298297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:18.304164 (+ 5867us) server_negotiation.cc:407] Connection header received
0929 15:52:18.304414 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:18.304419 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:18.304495 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:18.304597 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:52:18.305111 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:18.305993 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:18.307918 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:18.308360 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:18.308506 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:18.308974 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:52:18.309072 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:52:18.309171 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:18.309324 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 15:52:18.309416 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642}
I20260929 15:52:18.325084 8539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:18.319257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42035 (local address 127.0.97.2:38887)
0929 15:52:18.319413 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:18.319420 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:18.320028 (+ 608us) server_negotiation.cc:407] Connection header received
0929 15:52:18.320252 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:18.320257 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:18.320332 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:18.320423 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:52:18.320983 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:18.321850 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:18.323319 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:18.323735 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:18.323916 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:18.324332 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:52:18.324421 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:52:18.324515 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:18.324639 (+ 124us) server_negotiation.cc:299] Negotiation successful
0929 15:52:18.324704 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":48}
I20260929 15:52:18.325168 8547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:18.319236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38887 (local address 127.0.97.2:42035)
0929 15:52:18.319908 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:18.319928 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:18.320107 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:18.320451 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:18.320455 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:18.320483 (+ 28us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:52:18.320837 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:18.320846 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:18.321983 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:18.321990 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:18.323179 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:18.323190 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:18.323976 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:18.323984 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:18.324109 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:18.324153 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:52:18.324514 (+ 361us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:52:18.324524 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 15:52:18.324623 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 15:52:18.324746 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":20,"thread_start_us":388,"threads_started":1}
I20260929 15:52:18.333892 8423 raft_consensus.cc:1261] T d43a0825b39449db97017c7427f75e01 P 3a94f1c6bf464b549de59a4a1aecd307 [term 1 FOLLOWER]: Refusing update from remote peer b76aeb70d77b4a39b3243601bca4ffcb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:52:18.334028 8145 raft_consensus.cc:1261] T d43a0825b39449db97017c7427f75e01 P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 1 FOLLOWER]: Refusing update from remote peer b76aeb70d77b4a39b3243601bca4ffcb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:52:18.334532 8524 consensus_queue.cc:1035] T d43a0825b39449db97017c7427f75e01 P b76aeb70d77b4a39b3243601bca4ffcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:52:18.334998 8542 consensus_queue.cc:1035] T d43a0825b39449db97017c7427f75e01 P b76aeb70d77b4a39b3243601bca4ffcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:52:18.390605 7982 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:52:18.390902 7982 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:52:18.393692 7982 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 035d2b0927e444dc988dd4a5b7e816bd: Sending DeleteTablet for 3 replicas of tablet d43a0825b39449db97017c7427f75e01
I20260929 15:52:18.394375 8119 tablet_service.cc:1514] Processing DeleteTablet for tablet d43a0825b39449db97017c7427f75e01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:52:18 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39152
I20260929 15:52:18.394579 8398 tablet_service.cc:1514] Processing DeleteTablet for tablet d43a0825b39449db97017c7427f75e01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:52:18 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49876
I20260929 15:52:18.399767 8259 tablet_service.cc:1514] Processing DeleteTablet for tablet d43a0825b39449db97017c7427f75e01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:52:18 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48030
I20260929 15:52:18.404168 8553 tablet_replica.cc:330] T d43a0825b39449db97017c7427f75e01 P 3a94f1c6bf464b549de59a4a1aecd307: stopping tablet replica
I20260929 15:52:18.404505 8553 raft_consensus.cc:2229] T d43a0825b39449db97017c7427f75e01 P 3a94f1c6bf464b549de59a4a1aecd307 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:52:18.404822 8553 raft_consensus.cc:2258] T d43a0825b39449db97017c7427f75e01 P 3a94f1c6bf464b549de59a4a1aecd307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:52:18.406066 8553 ts_tablet_manager.cc:1905] T d43a0825b39449db97017c7427f75e01 P 3a94f1c6bf464b549de59a4a1aecd307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:52:18.411051 8554 tablet_replica.cc:330] T d43a0825b39449db97017c7427f75e01 P b76aeb70d77b4a39b3243601bca4ffcb: stopping tablet replica
I20260929 15:52:18.411289 8554 raft_consensus.cc:2229] T d43a0825b39449db97017c7427f75e01 P b76aeb70d77b4a39b3243601bca4ffcb [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:52:18.411505 8552 tablet_replica.cc:330] T d43a0825b39449db97017c7427f75e01 P 8bc1e5f5f2da4191ab74dbdb9e29af96: stopping tablet replica
I20260929 15:52:18.411726 8552 raft_consensus.cc:2229] T d43a0825b39449db97017c7427f75e01 P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:52:18.411855 8554 raft_consensus.cc:2258] T d43a0825b39449db97017c7427f75e01 P b76aeb70d77b4a39b3243601bca4ffcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:52:18.412154 8552 raft_consensus.cc:2258] T d43a0825b39449db97017c7427f75e01 P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:52:18.412760 8554 ts_tablet_manager.cc:1905] T d43a0825b39449db97017c7427f75e01 P b76aeb70d77b4a39b3243601bca4ffcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:52:18.414732 8552 ts_tablet_manager.cc:1905] T d43a0825b39449db97017c7427f75e01 P 8bc1e5f5f2da4191ab74dbdb9e29af96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:52:18.417701 8553 ts_tablet_manager.cc:1918] T d43a0825b39449db97017c7427f75e01 P 3a94f1c6bf464b549de59a4a1aecd307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:52:18.417872 8553 log.cc:1196] T d43a0825b39449db97017c7427f75e01 P 3a94f1c6bf464b549de59a4a1aecd307: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-2/wal/wals/d43a0825b39449db97017c7427f75e01
I20260929 15:52:18.418365 8553 ts_tablet_manager.cc:1939] T d43a0825b39449db97017c7427f75e01 P 3a94f1c6bf464b549de59a4a1aecd307: Deleting consensus metadata
I20260929 15:52:18.419481 7967 catalog_manager.cc:4841] TS 3a94f1c6bf464b549de59a4a1aecd307 (127.0.97.3:34959): tablet d43a0825b39449db97017c7427f75e01 (table test-table [id=c76db3435c26475dabe7db5b9e39a93f]) successfully deleted
I20260929 15:52:18.420073 8554 ts_tablet_manager.cc:1918] T d43a0825b39449db97017c7427f75e01 P b76aeb70d77b4a39b3243601bca4ffcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:52:18.420194 8554 log.cc:1196] T d43a0825b39449db97017c7427f75e01 P b76aeb70d77b4a39b3243601bca4ffcb: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-1/wal/wals/d43a0825b39449db97017c7427f75e01
I20260929 15:52:18.420608 8554 ts_tablet_manager.cc:1939] T d43a0825b39449db97017c7427f75e01 P b76aeb70d77b4a39b3243601bca4ffcb: Deleting consensus metadata
I20260929 15:52:18.421846 7967 catalog_manager.cc:4841] TS b76aeb70d77b4a39b3243601bca4ffcb (127.0.97.2:38887): tablet d43a0825b39449db97017c7427f75e01 (table test-table [id=c76db3435c26475dabe7db5b9e39a93f]) successfully deleted
I20260929 15:52:18.423904 8552 ts_tablet_manager.cc:1918] T d43a0825b39449db97017c7427f75e01 P 8bc1e5f5f2da4191ab74dbdb9e29af96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:52:18.424039 8552 log.cc:1196] T d43a0825b39449db97017c7427f75e01 P 8bc1e5f5f2da4191ab74dbdb9e29af96: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-0/wal/wals/d43a0825b39449db97017c7427f75e01
I20260929 15:52:18.424450 8552 ts_tablet_manager.cc:1939] T d43a0825b39449db97017c7427f75e01 P 8bc1e5f5f2da4191ab74dbdb9e29af96: Deleting consensus metadata
I20260929 15:52:18.425465 7967 catalog_manager.cc:4841] TS 8bc1e5f5f2da4191ab74dbdb9e29af96 (127.0.97.1:34987): tablet d43a0825b39449db97017c7427f75e01 (table test-table [id=c76db3435c26475dabe7db5b9e39a93f]) successfully deleted
Sep 29 15:52:18 dist-test-slave-r0ck krb5kdc[7938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 15:52:18 dist-test-slave-r0ck krb5kdc[7938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:18 dist-test-slave-r0ck krb5kdc[7938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 15:52:18.459275 8077 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:18.449221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60806 (local address 127.0.97.62:45367)
0929 15:52:18.449432 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:18.449439 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:18.449809 (+ 370us) server_negotiation.cc:407] Connection header received
0929 15:52:18.449962 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:18.449969 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:18.450049 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:18.450155 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:18.451300 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:18.452447 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:18.453667 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:18.454325 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:18.454454 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:18.459031 (+ 4577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60806: BlockingRecv error: recv got EOF from 127.0.0.1:60806 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 15:52:18 dist-test-slave-r0ck krb5kdc[7938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 15:52:18.475274 8077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:18.462086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60810 (local address 127.0.97.62:45367)
0929 15:52:18.462257 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:18.462264 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:18.462822 (+ 558us) server_negotiation.cc:407] Connection header received
0929 15:52:18.463014 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:18.463022 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:18.463103 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:18.463219 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:18.464325 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:18.465000 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:18.466301 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:18.466699 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:18.466823 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:18.470242 (+ 3419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:18.470247 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:18.470250 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:18.470278 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:18.472645 (+ 2367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:18.473231 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:18.473238 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:18.473241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:18.473435 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:18.473782 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:18.473789 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:18.473792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:18.474064 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:18.474314 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:18.474643 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 15:52:18.474899 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 15:52:18.478782 7982 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60810:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:52:18.479425 7982 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:52:18.486334 8119 tablet_service.cc:1467] Processing CreateTablet for tablet 7c8ea0303e844bc5a9e2fd7ff1c80cca (DEFAULT_TABLE table=test-table [id=6733c3c23f844248bea641210ae1818b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:52:18.486819 8119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c8ea0303e844bc5a9e2fd7ff1c80cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:52:18.486763 8259 tablet_service.cc:1467] Processing CreateTablet for tablet 7c8ea0303e844bc5a9e2fd7ff1c80cca (DEFAULT_TABLE table=test-table [id=6733c3c23f844248bea641210ae1818b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:52:18.486943 8398 tablet_service.cc:1467] Processing CreateTablet for tablet 7c8ea0303e844bc5a9e2fd7ff1c80cca (DEFAULT_TABLE table=test-table [id=6733c3c23f844248bea641210ae1818b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:52:18.487143 8259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c8ea0303e844bc5a9e2fd7ff1c80cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:52:18.487339 8398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c8ea0303e844bc5a9e2fd7ff1c80cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:52:18.493335 8493 tablet_bootstrap.cc:492] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 8bc1e5f5f2da4191ab74dbdb9e29af96: Bootstrap starting.
I20260929 15:52:18.494123 8492 tablet_bootstrap.cc:492] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P b76aeb70d77b4a39b3243601bca4ffcb: Bootstrap starting.
I20260929 15:52:18.494787 8494 tablet_bootstrap.cc:492] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 3a94f1c6bf464b549de59a4a1aecd307: Bootstrap starting.
I20260929 15:52:18.494993 8493 tablet_bootstrap.cc:654] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 8bc1e5f5f2da4191ab74dbdb9e29af96: Neither blocks nor log segments found. Creating new log.
I20260929 15:52:18.496549 8492 tablet_bootstrap.cc:654] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P b76aeb70d77b4a39b3243601bca4ffcb: Neither blocks nor log segments found. Creating new log.
I20260929 15:52:18.499469 8494 tablet_bootstrap.cc:654] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 3a94f1c6bf464b549de59a4a1aecd307: Neither blocks nor log segments found. Creating new log.
I20260929 15:52:18.504951 8493 tablet_bootstrap.cc:492] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 8bc1e5f5f2da4191ab74dbdb9e29af96: No bootstrap required, opened a new log
I20260929 15:52:18.505125 8493 ts_tablet_manager.cc:1397] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 8bc1e5f5f2da4191ab74dbdb9e29af96: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260929 15:52:18.505964 8493 raft_consensus.cc:348] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } }
I20260929 15:52:18.506222 8493 raft_consensus.cc:374] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:52:18.506331 8493 raft_consensus.cc:729] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bc1e5f5f2da4191ab74dbdb9e29af96, State: Initialized, Role: FOLLOWER
I20260929 15:52:18.506610 8493 consensus_queue.cc:260] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 8bc1e5f5f2da4191ab74dbdb9e29af96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } }
I20260929 15:52:18.507203 8493 ts_tablet_manager.cc:1428] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 8bc1e5f5f2da4191ab74dbdb9e29af96: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260929 15:52:18.511466 8492 tablet_bootstrap.cc:492] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P b76aeb70d77b4a39b3243601bca4ffcb: No bootstrap required, opened a new log
I20260929 15:52:18.511638 8492 ts_tablet_manager.cc:1397] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P b76aeb70d77b4a39b3243601bca4ffcb: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.004s
I20260929 15:52:18.511739 8494 tablet_bootstrap.cc:492] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 3a94f1c6bf464b549de59a4a1aecd307: No bootstrap required, opened a new log
I20260929 15:52:18.511893 8494 ts_tablet_manager.cc:1397] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 3a94f1c6bf464b549de59a4a1aecd307: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.007s
I20260929 15:52:18.512326 8492 raft_consensus.cc:348] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P b76aeb70d77b4a39b3243601bca4ffcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } }
I20260929 15:52:18.512557 8492 raft_consensus.cc:374] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P b76aeb70d77b4a39b3243601bca4ffcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:52:18.512672 8492 raft_consensus.cc:729] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P b76aeb70d77b4a39b3243601bca4ffcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b76aeb70d77b4a39b3243601bca4ffcb, State: Initialized, Role: FOLLOWER
I20260929 15:52:18.512579 8494 raft_consensus.cc:348] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 3a94f1c6bf464b549de59a4a1aecd307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } }
I20260929 15:52:18.512841 8494 raft_consensus.cc:374] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 3a94f1c6bf464b549de59a4a1aecd307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:52:18.512930 8494 raft_consensus.cc:729] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 3a94f1c6bf464b549de59a4a1aecd307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a94f1c6bf464b549de59a4a1aecd307, State: Initialized, Role: FOLLOWER
I20260929 15:52:18.512928 8492 consensus_queue.cc:260] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P b76aeb70d77b4a39b3243601bca4ffcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } }
I20260929 15:52:18.513536 8492 ts_tablet_manager.cc:1428] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P b76aeb70d77b4a39b3243601bca4ffcb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 15:52:18.513163 8494 consensus_queue.cc:260] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 3a94f1c6bf464b549de59a4a1aecd307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } }
I20260929 15:52:18.514778 8494 ts_tablet_manager.cc:1428] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 3a94f1c6bf464b549de59a4a1aecd307: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260929 15:52:18.540920 8503 raft_consensus.cc:482] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:52:18.541121 8503 raft_consensus.cc:504] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } }
I20260929 15:52:18.541733 8503 leader_election.cc:288] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 8bc1e5f5f2da4191ab74dbdb9e29af96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b76aeb70d77b4a39b3243601bca4ffcb (127.0.97.2:38887), 3a94f1c6bf464b549de59a4a1aecd307 (127.0.97.3:34959)
I20260929 15:52:18.542332 8423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8ea0303e844bc5a9e2fd7ff1c80cca" candidate_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a94f1c6bf464b549de59a4a1aecd307" is_pre_election: true
I20260929 15:52:18.542352 8279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8ea0303e844bc5a9e2fd7ff1c80cca" candidate_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" is_pre_election: true
I20260929 15:52:18.542567 8423 raft_consensus.cc:2454] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 3a94f1c6bf464b549de59a4a1aecd307 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bc1e5f5f2da4191ab74dbdb9e29af96 in term 0.
I20260929 15:52:18.542632 8279 raft_consensus.cc:2454] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P b76aeb70d77b4a39b3243601bca4ffcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bc1e5f5f2da4191ab74dbdb9e29af96 in term 0.
I20260929 15:52:18.543035 8070 leader_election.cc:302] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 8bc1e5f5f2da4191ab74dbdb9e29af96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a94f1c6bf464b549de59a4a1aecd307, 8bc1e5f5f2da4191ab74dbdb9e29af96; no voters:
I20260929 15:52:18.543313 8503 raft_consensus.cc:2781] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:52:18.543429 8503 raft_consensus.cc:482] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:52:18.543547 8503 raft_consensus.cc:3037] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:52:18.545524 8503 raft_consensus.cc:504] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } }
I20260929 15:52:18.546088 8503 leader_election.cc:288] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 8bc1e5f5f2da4191ab74dbdb9e29af96 [CANDIDATE]: Term 1 election: Requested vote from peers b76aeb70d77b4a39b3243601bca4ffcb (127.0.97.2:38887), 3a94f1c6bf464b549de59a4a1aecd307 (127.0.97.3:34959)
I20260929 15:52:18.546600 8279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8ea0303e844bc5a9e2fd7ff1c80cca" candidate_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b76aeb70d77b4a39b3243601bca4ffcb"
I20260929 15:52:18.546813 8279 raft_consensus.cc:3037] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P b76aeb70d77b4a39b3243601bca4ffcb [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:52:18.546772 8423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8ea0303e844bc5a9e2fd7ff1c80cca" candidate_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a94f1c6bf464b549de59a4a1aecd307"
I20260929 15:52:18.546975 8423 raft_consensus.cc:3037] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 3a94f1c6bf464b549de59a4a1aecd307 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:52:18.549933 8279 raft_consensus.cc:2454] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P b76aeb70d77b4a39b3243601bca4ffcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bc1e5f5f2da4191ab74dbdb9e29af96 in term 1.
I20260929 15:52:18.550310 8070 leader_election.cc:302] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 8bc1e5f5f2da4191ab74dbdb9e29af96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bc1e5f5f2da4191ab74dbdb9e29af96, b76aeb70d77b4a39b3243601bca4ffcb; no voters:
I20260929 15:52:18.550499 8423 raft_consensus.cc:2454] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 3a94f1c6bf464b549de59a4a1aecd307 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bc1e5f5f2da4191ab74dbdb9e29af96 in term 1.
I20260929 15:52:18.550563 8503 raft_consensus.cc:2781] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:52:18.551345 8503 raft_consensus.cc:686] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 1 LEADER]: Becoming Leader. State: Replica: 8bc1e5f5f2da4191ab74dbdb9e29af96, State: Running, Role: LEADER
I20260929 15:52:18.551923 8503 consensus_queue.cc:237] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 8bc1e5f5f2da4191ab74dbdb9e29af96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } }
I20260929 15:52:18.554359 7983 catalog_manager.cc:5495] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 8bc1e5f5f2da4191ab74dbdb9e29af96 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8bc1e5f5f2da4191ab74dbdb9e29af96 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } health_report { overall_health: UNKNOWN } } }
I20260929 15:52:18.602123 8573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:18.584136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39170 (local address 127.0.97.1:34987)
0929 15:52:18.595985 (+ 11849us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:18.595991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:18.596012 (+ 21us) server_negotiation.cc:407] Connection header received
0929 15:52:18.596080 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:18.596085 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:18.596189 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:18.596299 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:52:18.596771 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:18.597974 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:18.599976 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:18.600696 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:18.600822 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:18.601307 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:52:18.601449 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:52:18.601530 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:18.601695 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 15:52:18.601777 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11716,"thread_start_us":11556,"threads_started":1}
I20260929 15:52:18.604911 8423 raft_consensus.cc:1261] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 3a94f1c6bf464b549de59a4a1aecd307 [term 1 FOLLOWER]: Refusing update from remote peer 8bc1e5f5f2da4191ab74dbdb9e29af96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:52:18.604983 8279 raft_consensus.cc:1261] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P b76aeb70d77b4a39b3243601bca4ffcb [term 1 FOLLOWER]: Refusing update from remote peer 8bc1e5f5f2da4191ab74dbdb9e29af96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:52:18.605548 8503 consensus_queue.cc:1035] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 8bc1e5f5f2da4191ab74dbdb9e29af96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:52:18.605835 8570 consensus_queue.cc:1035] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 8bc1e5f5f2da4191ab74dbdb9e29af96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:52:18.618871 7982 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:52:18.619072 7982 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:52:18.624097 7982 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 035d2b0927e444dc988dd4a5b7e816bd: Sending DeleteTablet for 3 replicas of tablet 7c8ea0303e844bc5a9e2fd7ff1c80cca
I20260929 15:52:18.625139 8259 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c8ea0303e844bc5a9e2fd7ff1c80cca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:52:18 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48030
I20260929 15:52:18.625368 8119 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c8ea0303e844bc5a9e2fd7ff1c80cca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:52:18 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39152
I20260929 15:52:18.625488 8554 tablet_replica.cc:330] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P b76aeb70d77b4a39b3243601bca4ffcb: stopping tablet replica
I20260929 15:52:18.625602 8552 tablet_replica.cc:330] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 8bc1e5f5f2da4191ab74dbdb9e29af96: stopping tablet replica
I20260929 15:52:18.625622 8398 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c8ea0303e844bc5a9e2fd7ff1c80cca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:52:18 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49876
I20260929 15:52:18.625707 8554 raft_consensus.cc:2229] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P b76aeb70d77b4a39b3243601bca4ffcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:52:18.625788 8552 raft_consensus.cc:2229] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:52:18.625890 8553 tablet_replica.cc:330] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 3a94f1c6bf464b549de59a4a1aecd307: stopping tablet replica
I20260929 15:52:18.625908 8554 raft_consensus.cc:2258] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P b76aeb70d77b4a39b3243601bca4ffcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:52:18.626051 8552 raft_consensus.cc:2258] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:52:18.626065 8553 raft_consensus.cc:2229] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 3a94f1c6bf464b549de59a4a1aecd307 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:52:18.626286 8553 raft_consensus.cc:2258] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 3a94f1c6bf464b549de59a4a1aecd307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:52:18.625485 8258 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c8ea0303e844bc5a9e2fd7ff1c80cca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:52:18 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48030
I20260929 15:52:18.627537 8553 ts_tablet_manager.cc:1905] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 3a94f1c6bf464b549de59a4a1aecd307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:52:18.628095 8552 ts_tablet_manager.cc:1905] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 8bc1e5f5f2da4191ab74dbdb9e29af96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:52:18.630622 7981 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60810:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:52:18.631316 7981 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:52:18.634261 8553 ts_tablet_manager.cc:1918] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 3a94f1c6bf464b549de59a4a1aecd307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:52:18.634418 8553 log.cc:1196] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 3a94f1c6bf464b549de59a4a1aecd307: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-2/wal/wals/7c8ea0303e844bc5a9e2fd7ff1c80cca
I20260929 15:52:18.635160 8552 ts_tablet_manager.cc:1918] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 8bc1e5f5f2da4191ab74dbdb9e29af96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:52:18.635316 8552 log.cc:1196] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 8bc1e5f5f2da4191ab74dbdb9e29af96: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-0/wal/wals/7c8ea0303e844bc5a9e2fd7ff1c80cca
I20260929 15:52:18.636073 8553 ts_tablet_manager.cc:1939] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 3a94f1c6bf464b549de59a4a1aecd307: Deleting consensus metadata
I20260929 15:52:18.636343 8552 ts_tablet_manager.cc:1939] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P 8bc1e5f5f2da4191ab74dbdb9e29af96: Deleting consensus metadata
I20260929 15:52:18.637102 7967 catalog_manager.cc:4841] TS 3a94f1c6bf464b549de59a4a1aecd307 (127.0.97.3:34959): tablet 7c8ea0303e844bc5a9e2fd7ff1c80cca (table test-table [id=6733c3c23f844248bea641210ae1818b]) successfully deleted
I20260929 15:52:18.637362 7967 catalog_manager.cc:4841] TS 8bc1e5f5f2da4191ab74dbdb9e29af96 (127.0.97.1:34987): tablet 7c8ea0303e844bc5a9e2fd7ff1c80cca (table test-table [id=6733c3c23f844248bea641210ae1818b]) successfully deleted
I20260929 15:52:18.638887 8258 tablet_service.cc:1467] Processing CreateTablet for tablet 5a9a05fdeb694d83abc70da516b09e0b (DEFAULT_TABLE table=test-table [id=ac22a3c94f69477dab5c8cbd1aac1bd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:52:18.639338 8258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a9a05fdeb694d83abc70da516b09e0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:52:18.646047 8554 ts_tablet_manager.cc:1905] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P b76aeb70d77b4a39b3243601bca4ffcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:52:18.646153 8398 tablet_service.cc:1467] Processing CreateTablet for tablet 5a9a05fdeb694d83abc70da516b09e0b (DEFAULT_TABLE table=test-table [id=ac22a3c94f69477dab5c8cbd1aac1bd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:52:18.646373 8119 tablet_service.cc:1467] Processing CreateTablet for tablet 5a9a05fdeb694d83abc70da516b09e0b (DEFAULT_TABLE table=test-table [id=ac22a3c94f69477dab5c8cbd1aac1bd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:52:18.646557 8398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a9a05fdeb694d83abc70da516b09e0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:52:18.646777 8119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a9a05fdeb694d83abc70da516b09e0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:52:18.652872 8494 tablet_bootstrap.cc:492] T 5a9a05fdeb694d83abc70da516b09e0b P 3a94f1c6bf464b549de59a4a1aecd307: Bootstrap starting.
I20260929 15:52:18.653869 8493 tablet_bootstrap.cc:492] T 5a9a05fdeb694d83abc70da516b09e0b P 8bc1e5f5f2da4191ab74dbdb9e29af96: Bootstrap starting.
I20260929 15:52:18.653481 8554 ts_tablet_manager.cc:1918] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P b76aeb70d77b4a39b3243601bca4ffcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:52:18.654590 8494 tablet_bootstrap.cc:654] T 5a9a05fdeb694d83abc70da516b09e0b P 3a94f1c6bf464b549de59a4a1aecd307: Neither blocks nor log segments found. Creating new log.
I20260929 15:52:18.654585 8554 log.cc:1196] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P b76aeb70d77b4a39b3243601bca4ffcb: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-1/wal/wals/7c8ea0303e844bc5a9e2fd7ff1c80cca
I20260929 15:52:18.655069 8554 ts_tablet_manager.cc:1939] T 7c8ea0303e844bc5a9e2fd7ff1c80cca P b76aeb70d77b4a39b3243601bca4ffcb: Deleting consensus metadata
I20260929 15:52:18.655515 8493 tablet_bootstrap.cc:654] T 5a9a05fdeb694d83abc70da516b09e0b P 8bc1e5f5f2da4191ab74dbdb9e29af96: Neither blocks nor log segments found. Creating new log.
I20260929 15:52:18.656534 7967 catalog_manager.cc:4841] TS b76aeb70d77b4a39b3243601bca4ffcb (127.0.97.2:38887): tablet 7c8ea0303e844bc5a9e2fd7ff1c80cca (table test-table [id=6733c3c23f844248bea641210ae1818b]) successfully deleted
W20260929 15:52:18.657658 7967 catalog_manager.cc:4808] TS b76aeb70d77b4a39b3243601bca4ffcb (127.0.97.2:38887): delete failed for tablet 7c8ea0303e844bc5a9e2fd7ff1c80cca because the tablet was not found. No further retry: Not found: Tablet not found: 7c8ea0303e844bc5a9e2fd7ff1c80cca
I20260929 15:52:18.659636 8492 tablet_bootstrap.cc:492] T 5a9a05fdeb694d83abc70da516b09e0b P b76aeb70d77b4a39b3243601bca4ffcb: Bootstrap starting.
I20260929 15:52:18.662071 8492 tablet_bootstrap.cc:654] T 5a9a05fdeb694d83abc70da516b09e0b P b76aeb70d77b4a39b3243601bca4ffcb: Neither blocks nor log segments found. Creating new log.
I20260929 15:52:18.664491 8493 tablet_bootstrap.cc:492] T 5a9a05fdeb694d83abc70da516b09e0b P 8bc1e5f5f2da4191ab74dbdb9e29af96: No bootstrap required, opened a new log
I20260929 15:52:18.664600 8492 tablet_bootstrap.cc:492] T 5a9a05fdeb694d83abc70da516b09e0b P b76aeb70d77b4a39b3243601bca4ffcb: No bootstrap required, opened a new log
I20260929 15:52:18.664644 8493 ts_tablet_manager.cc:1397] T 5a9a05fdeb694d83abc70da516b09e0b P 8bc1e5f5f2da4191ab74dbdb9e29af96: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20260929 15:52:18.664759 8492 ts_tablet_manager.cc:1397] T 5a9a05fdeb694d83abc70da516b09e0b P b76aeb70d77b4a39b3243601bca4ffcb: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 15:52:18.665390 8493 raft_consensus.cc:348] T 5a9a05fdeb694d83abc70da516b09e0b P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } }
I20260929 15:52:18.665643 8493 raft_consensus.cc:374] T 5a9a05fdeb694d83abc70da516b09e0b P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:52:18.665729 8493 raft_consensus.cc:729] T 5a9a05fdeb694d83abc70da516b09e0b P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bc1e5f5f2da4191ab74dbdb9e29af96, State: Initialized, Role: FOLLOWER
I20260929 15:52:18.665575 8492 raft_consensus.cc:348] T 5a9a05fdeb694d83abc70da516b09e0b P b76aeb70d77b4a39b3243601bca4ffcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } }
I20260929 15:52:18.665853 8492 raft_consensus.cc:374] T 5a9a05fdeb694d83abc70da516b09e0b P b76aeb70d77b4a39b3243601bca4ffcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:52:18.665954 8492 raft_consensus.cc:729] T 5a9a05fdeb694d83abc70da516b09e0b P b76aeb70d77b4a39b3243601bca4ffcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b76aeb70d77b4a39b3243601bca4ffcb, State: Initialized, Role: FOLLOWER
I20260929 15:52:18.665984 8493 consensus_queue.cc:260] T 5a9a05fdeb694d83abc70da516b09e0b P 8bc1e5f5f2da4191ab74dbdb9e29af96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } }
I20260929 15:52:18.666211 8492 consensus_queue.cc:260] T 5a9a05fdeb694d83abc70da516b09e0b P b76aeb70d77b4a39b3243601bca4ffcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } }
I20260929 15:52:18.666622 8493 ts_tablet_manager.cc:1428] T 5a9a05fdeb694d83abc70da516b09e0b P 8bc1e5f5f2da4191ab74dbdb9e29af96: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 15:52:18.667259 8492 ts_tablet_manager.cc:1428] T 5a9a05fdeb694d83abc70da516b09e0b P b76aeb70d77b4a39b3243601bca4ffcb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 15:52:18.671185 8570 raft_consensus.cc:482] T 5a9a05fdeb694d83abc70da516b09e0b P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:52:18.671367 8570 raft_consensus.cc:504] T 5a9a05fdeb694d83abc70da516b09e0b P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } }
I20260929 15:52:18.672051 8570 leader_election.cc:288] T 5a9a05fdeb694d83abc70da516b09e0b P 8bc1e5f5f2da4191ab74dbdb9e29af96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b76aeb70d77b4a39b3243601bca4ffcb (127.0.97.2:38887), 3a94f1c6bf464b549de59a4a1aecd307 (127.0.97.3:34959)
I20260929 15:52:18.672179 8494 tablet_bootstrap.cc:492] T 5a9a05fdeb694d83abc70da516b09e0b P 3a94f1c6bf464b549de59a4a1aecd307: No bootstrap required, opened a new log
I20260929 15:52:18.672348 8494 ts_tablet_manager.cc:1397] T 5a9a05fdeb694d83abc70da516b09e0b P 3a94f1c6bf464b549de59a4a1aecd307: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.002s
I20260929 15:52:18.672587 8423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a9a05fdeb694d83abc70da516b09e0b" candidate_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a94f1c6bf464b549de59a4a1aecd307" is_pre_election: true
I20260929 15:52:18.673223 8494 raft_consensus.cc:348] T 5a9a05fdeb694d83abc70da516b09e0b P 3a94f1c6bf464b549de59a4a1aecd307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } }
I20260929 15:52:18.673514 8494 raft_consensus.cc:374] T 5a9a05fdeb694d83abc70da516b09e0b P 3a94f1c6bf464b549de59a4a1aecd307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:52:18.673605 8494 raft_consensus.cc:729] T 5a9a05fdeb694d83abc70da516b09e0b P 3a94f1c6bf464b549de59a4a1aecd307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a94f1c6bf464b549de59a4a1aecd307, State: Initialized, Role: FOLLOWER
W20260929 15:52:18.673568 8070 leader_election.cc:341] T 5a9a05fdeb694d83abc70da516b09e0b P 8bc1e5f5f2da4191ab74dbdb9e29af96 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3a94f1c6bf464b549de59a4a1aecd307 (127.0.97.3:34959): Illegal state: must be running to vote when last-logged opid is not known
I20260929 15:52:18.673906 8279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a9a05fdeb694d83abc70da516b09e0b" candidate_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" is_pre_election: true
I20260929 15:52:18.673883 8494 consensus_queue.cc:260] T 5a9a05fdeb694d83abc70da516b09e0b P 3a94f1c6bf464b549de59a4a1aecd307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } }
I20260929 15:52:18.674158 8279 raft_consensus.cc:2454] T 5a9a05fdeb694d83abc70da516b09e0b P b76aeb70d77b4a39b3243601bca4ffcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bc1e5f5f2da4191ab74dbdb9e29af96 in term 0.
I20260929 15:52:18.674525 8070 leader_election.cc:302] T 5a9a05fdeb694d83abc70da516b09e0b P 8bc1e5f5f2da4191ab74dbdb9e29af96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8bc1e5f5f2da4191ab74dbdb9e29af96, b76aeb70d77b4a39b3243601bca4ffcb; no voters: 3a94f1c6bf464b549de59a4a1aecd307
I20260929 15:52:18.674981 8570 raft_consensus.cc:2781] T 5a9a05fdeb694d83abc70da516b09e0b P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:52:18.675010 8494 ts_tablet_manager.cc:1428] T 5a9a05fdeb694d83abc70da516b09e0b P 3a94f1c6bf464b549de59a4a1aecd307: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 15:52:18.675084 8570 raft_consensus.cc:482] T 5a9a05fdeb694d83abc70da516b09e0b P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:52:18.675215 8570 raft_consensus.cc:3037] T 5a9a05fdeb694d83abc70da516b09e0b P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:52:18.677860 8570 raft_consensus.cc:504] T 5a9a05fdeb694d83abc70da516b09e0b P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } }
I20260929 15:52:18.678435 8570 leader_election.cc:288] T 5a9a05fdeb694d83abc70da516b09e0b P 8bc1e5f5f2da4191ab74dbdb9e29af96 [CANDIDATE]: Term 1 election: Requested vote from peers b76aeb70d77b4a39b3243601bca4ffcb (127.0.97.2:38887), 3a94f1c6bf464b549de59a4a1aecd307 (127.0.97.3:34959)
I20260929 15:52:18.679054 8423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a9a05fdeb694d83abc70da516b09e0b" candidate_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a94f1c6bf464b549de59a4a1aecd307"
I20260929 15:52:18.679090 8279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a9a05fdeb694d83abc70da516b09e0b" candidate_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b76aeb70d77b4a39b3243601bca4ffcb"
I20260929 15:52:18.679278 8279 raft_consensus.cc:3037] T 5a9a05fdeb694d83abc70da516b09e0b P b76aeb70d77b4a39b3243601bca4ffcb [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:52:18.679286 8423 raft_consensus.cc:3037] T 5a9a05fdeb694d83abc70da516b09e0b P 3a94f1c6bf464b549de59a4a1aecd307 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:52:18.682150 8423 raft_consensus.cc:2454] T 5a9a05fdeb694d83abc70da516b09e0b P 3a94f1c6bf464b549de59a4a1aecd307 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bc1e5f5f2da4191ab74dbdb9e29af96 in term 1.
I20260929 15:52:18.682349 8279 raft_consensus.cc:2454] T 5a9a05fdeb694d83abc70da516b09e0b P b76aeb70d77b4a39b3243601bca4ffcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bc1e5f5f2da4191ab74dbdb9e29af96 in term 1.
I20260929 15:52:18.682552 8070 leader_election.cc:302] T 5a9a05fdeb694d83abc70da516b09e0b P 8bc1e5f5f2da4191ab74dbdb9e29af96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a94f1c6bf464b549de59a4a1aecd307, 8bc1e5f5f2da4191ab74dbdb9e29af96; no voters:
I20260929 15:52:18.682811 8570 raft_consensus.cc:2781] T 5a9a05fdeb694d83abc70da516b09e0b P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:52:18.682991 8570 raft_consensus.cc:686] T 5a9a05fdeb694d83abc70da516b09e0b P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 1 LEADER]: Becoming Leader. State: Replica: 8bc1e5f5f2da4191ab74dbdb9e29af96, State: Running, Role: LEADER
I20260929 15:52:18.683254 8570 consensus_queue.cc:237] T 5a9a05fdeb694d83abc70da516b09e0b P 8bc1e5f5f2da4191ab74dbdb9e29af96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } } peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } }
I20260929 15:52:18.685817 7982 catalog_manager.cc:5495] T 5a9a05fdeb694d83abc70da516b09e0b P 8bc1e5f5f2da4191ab74dbdb9e29af96 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8bc1e5f5f2da4191ab74dbdb9e29af96 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8bc1e5f5f2da4191ab74dbdb9e29af96" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 } health_report { overall_health: UNKNOWN } } }
I20260929 15:52:18.718155 8539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:18.712514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42631 (local address 127.0.97.2:38887)
0929 15:52:18.712689 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:18.712693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:18.713178 (+ 485us) server_negotiation.cc:407] Connection header received
0929 15:52:18.713398 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:18.713404 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:18.713477 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:18.713568 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:52:18.714050 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:18.714889 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:18.716338 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:18.716818 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:18.716976 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:18.717394 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:52:18.717513 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:52:18.717615 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:18.717774 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 15:52:18.717846 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 15:52:18.718263 8582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:18.712548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38887 (local address 127.0.97.1:42631)
0929 15:52:18.713055 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:18.713093 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:18.713246 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:18.713594 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:18.713599 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:18.713610 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:52:18.713892 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:18.713901 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:18.715022 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:18.715027 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:18.716198 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:18.716210 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:18.716983 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:18.716988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:18.717123 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:18.717172 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:52:18.717612 (+ 440us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:52:18.717632 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 15:52:18.717749 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 15:52:18.717871 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":271,"threads_started":1}
I20260929 15:52:18.725975 8423 raft_consensus.cc:1261] T 5a9a05fdeb694d83abc70da516b09e0b P 3a94f1c6bf464b549de59a4a1aecd307 [term 1 FOLLOWER]: Refusing update from remote peer 8bc1e5f5f2da4191ab74dbdb9e29af96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:52:18.726523 8279 raft_consensus.cc:1261] T 5a9a05fdeb694d83abc70da516b09e0b P b76aeb70d77b4a39b3243601bca4ffcb [term 1 FOLLOWER]: Refusing update from remote peer 8bc1e5f5f2da4191ab74dbdb9e29af96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:52:18.727005 8503 consensus_queue.cc:1035] T 5a9a05fdeb694d83abc70da516b09e0b P 8bc1e5f5f2da4191ab74dbdb9e29af96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b76aeb70d77b4a39b3243601bca4ffcb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:52:18.727258 8570 consensus_queue.cc:1035] T 5a9a05fdeb694d83abc70da516b09e0b P 8bc1e5f5f2da4191ab74dbdb9e29af96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a94f1c6bf464b549de59a4a1aecd307" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:52:18.753156 8547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:18.746821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34987 (local address 127.0.97.2:51251)
0929 15:52:18.747040 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:18.747062 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:18.747167 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:18.747946 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:18.747951 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:18.747965 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:52:18.748155 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:18.748166 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:18.749615 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:18.749621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:18.751286 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:18.751299 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:18.752169 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:18.752172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:18.752290 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:18.752321 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:52:18.752693 (+ 372us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:52:18.752700 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 15:52:18.752759 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 15:52:18.752851 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 15:52:18.753170 8573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:18.747359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51251 (local address 127.0.97.1:34987)
0929 15:52:18.747529 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:18.747537 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:18.747555 (+ 18us) server_negotiation.cc:407] Connection header received
0929 15:52:18.747611 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:18.747616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:18.747690 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:18.747775 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:52:18.748325 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:18.749464 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:18.751412 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:18.752027 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:18.752151 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:18.752470 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:52:18.752596 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:52:18.752690 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:18.752793 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 15:52:18.752864 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 15:52:18.776777 7982 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:52:18.776963 7982 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:52:18.779696 7982 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 035d2b0927e444dc988dd4a5b7e816bd: Sending DeleteTablet for 3 replicas of tablet 5a9a05fdeb694d83abc70da516b09e0b
I20260929 15:52:18.780519 8398 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a9a05fdeb694d83abc70da516b09e0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:52:18 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49876
I20260929 15:52:18.780550 8119 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a9a05fdeb694d83abc70da516b09e0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:52:18 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39152
I20260929 15:52:18.780798 8553 tablet_replica.cc:330] T 5a9a05fdeb694d83abc70da516b09e0b P 3a94f1c6bf464b549de59a4a1aecd307: stopping tablet replica
I20260929 15:52:18.780800 8552 tablet_replica.cc:330] T 5a9a05fdeb694d83abc70da516b09e0b P 8bc1e5f5f2da4191ab74dbdb9e29af96: stopping tablet replica
I20260929 15:52:18.780977 8553 raft_consensus.cc:2229] T 5a9a05fdeb694d83abc70da516b09e0b P 3a94f1c6bf464b549de59a4a1aecd307 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:52:18.780975 8552 raft_consensus.cc:2229] T 5a9a05fdeb694d83abc70da516b09e0b P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:52:18.781175 8553 raft_consensus.cc:2258] T 5a9a05fdeb694d83abc70da516b09e0b P 3a94f1c6bf464b549de59a4a1aecd307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:52:18.781277 8552 raft_consensus.cc:2258] T 5a9a05fdeb694d83abc70da516b09e0b P 8bc1e5f5f2da4191ab74dbdb9e29af96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:52:18.782192 8553 ts_tablet_manager.cc:1905] T 5a9a05fdeb694d83abc70da516b09e0b P 3a94f1c6bf464b549de59a4a1aecd307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:52:18.782157 8258 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a9a05fdeb694d83abc70da516b09e0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:52:18 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48030
I20260929 15:52:18.782282 8552 ts_tablet_manager.cc:1905] T 5a9a05fdeb694d83abc70da516b09e0b P 8bc1e5f5f2da4191ab74dbdb9e29af96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:52:18.787042 8553 ts_tablet_manager.cc:1918] T 5a9a05fdeb694d83abc70da516b09e0b P 3a94f1c6bf464b549de59a4a1aecd307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:52:18.787124 8553 log.cc:1196] T 5a9a05fdeb694d83abc70da516b09e0b P 3a94f1c6bf464b549de59a4a1aecd307: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-2/wal/wals/5a9a05fdeb694d83abc70da516b09e0b
I20260929 15:52:18.787408 8553 ts_tablet_manager.cc:1939] T 5a9a05fdeb694d83abc70da516b09e0b P 3a94f1c6bf464b549de59a4a1aecd307: Deleting consensus metadata
I20260929 15:52:18.788163 7967 catalog_manager.cc:4841] TS 3a94f1c6bf464b549de59a4a1aecd307 (127.0.97.3:34959): tablet 5a9a05fdeb694d83abc70da516b09e0b (table test-table [id=ac22a3c94f69477dab5c8cbd1aac1bd9]) successfully deleted
I20260929 15:52:18.788291 8554 tablet_replica.cc:330] T 5a9a05fdeb694d83abc70da516b09e0b P b76aeb70d77b4a39b3243601bca4ffcb: stopping tablet replica
I20260929 15:52:18.788447 8554 raft_consensus.cc:2229] T 5a9a05fdeb694d83abc70da516b09e0b P b76aeb70d77b4a39b3243601bca4ffcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:52:18.788591 8554 raft_consensus.cc:2258] T 5a9a05fdeb694d83abc70da516b09e0b P b76aeb70d77b4a39b3243601bca4ffcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:52:18.788770 8552 ts_tablet_manager.cc:1918] T 5a9a05fdeb694d83abc70da516b09e0b P 8bc1e5f5f2da4191ab74dbdb9e29af96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:52:18.788887 8552 log.cc:1196] T 5a9a05fdeb694d83abc70da516b09e0b P 8bc1e5f5f2da4191ab74dbdb9e29af96: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-0/wal/wals/5a9a05fdeb694d83abc70da516b09e0b
I20260929 15:52:18.789321 8552 ts_tablet_manager.cc:1939] T 5a9a05fdeb694d83abc70da516b09e0b P 8bc1e5f5f2da4191ab74dbdb9e29af96: Deleting consensus metadata
I20260929 15:52:18.789531 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 8054
I20260929 15:52:18.790959 7967 catalog_manager.cc:4841] TS 8bc1e5f5f2da4191ab74dbdb9e29af96 (127.0.97.1:34987): tablet 5a9a05fdeb694d83abc70da516b09e0b (table test-table [id=ac22a3c94f69477dab5c8cbd1aac1bd9]) successfully deleted
I20260929 15:52:18.791798 8554 ts_tablet_manager.cc:1905] T 5a9a05fdeb694d83abc70da516b09e0b P b76aeb70d77b4a39b3243601bca4ffcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:52:18.797452 8554 ts_tablet_manager.cc:1918] T 5a9a05fdeb694d83abc70da516b09e0b P b76aeb70d77b4a39b3243601bca4ffcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:52:18.797607 8554 log.cc:1196] T 5a9a05fdeb694d83abc70da516b09e0b P b76aeb70d77b4a39b3243601bca4ffcb: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697075657932-388-0/minicluster-data/ts-1/wal/wals/5a9a05fdeb694d83abc70da516b09e0b
I20260929 15:52:18.798002 8554 ts_tablet_manager.cc:1939] T 5a9a05fdeb694d83abc70da516b09e0b P b76aeb70d77b4a39b3243601bca4ffcb: Deleting consensus metadata
I20260929 15:52:18.798797 7967 catalog_manager.cc:4841] TS b76aeb70d77b4a39b3243601bca4ffcb (127.0.97.2:38887): tablet 5a9a05fdeb694d83abc70da516b09e0b (table test-table [id=ac22a3c94f69477dab5c8cbd1aac1bd9]) successfully deleted
Sep 29 15:52:18 dist-test-slave-r0ck krb5kdc[7938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260929 15:52:18.806785 8573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:18.790618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39184 (local address 127.0.97.1:34987)
0929 15:52:18.790780 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:18.790786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:18.790959 (+ 173us) server_negotiation.cc:407] Connection header received
0929 15:52:18.791257 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:18.791263 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:18.791343 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:18.791415 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:18.793065 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:18.794132 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:18.795921 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:18.796591 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:18.796703 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:18.801371 (+ 4668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:18.801386 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:18.801405 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:18.801445 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:18.804242 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:18.804845 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:18.804853 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:18.804857 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:18.804986 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:18.805368 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:18.805373 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:18.805377 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:18.805670 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:18.805980 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:18.806249 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 15:52:18.806449 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 15:52:18.807374 8185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:52:18.931221 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 8054
I20260929 15:52:18.956079 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 8194
Sep 29 15:52:18 dist-test-slave-r0ck krb5kdc[7938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260929 15:52:18.975416 8539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:18.956822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48042 (local address 127.0.97.2:38887)
0929 15:52:18.957016 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:18.957022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:18.957043 (+ 21us) server_negotiation.cc:407] Connection header received
0929 15:52:18.957209 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:18.957214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:18.957303 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:18.957432 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:18.959130 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:18.960300 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:18.962131 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:18.962869 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:18.962988 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:18.968108 (+ 5120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:18.968124 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:18.968151 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:18.968192 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:18.972374 (+ 4182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:18.973125 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:18.973134 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:18.973137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:18.973298 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:18.973691 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:18.973695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:18.973699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:18.974148 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:18.974414 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:18.974778 (+ 364us) server_negotiation.cc:299] Negotiation successful
0929 15:52:18.974999 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 15:52:18.976367 8324 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:52:19.121780 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 8194
I20260929 15:52:19.155848 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 8333
Sep 29 15:52:19 dist-test-slave-r0ck krb5kdc[7938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260929 15:52:19.172830 8589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:19.156680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49880 (local address 127.0.97.3:34959)
0929 15:52:19.157182 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:19.157186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:19.157202 (+ 16us) server_negotiation.cc:407] Connection header received
0929 15:52:19.157255 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:19.157258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:19.157316 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:19.157474 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:19.158713 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:19.160050 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:19.161228 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:19.162036 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:19.162180 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:19.167854 (+ 5674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:19.167868 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:19.167889 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:19.167918 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:19.170634 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:19.171209 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:19.171215 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:19.171218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:19.171340 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:19.171640 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:19.171643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:19.171645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:19.171942 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:19.172138 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:19.172420 (+ 282us) server_negotiation.cc:299] Negotiation successful
0929 15:52:19.172564 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":262,"threads_started":1}
I20260929 15:52:19.173563 8463 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:52:19.268201 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 8333
I20260929 15:52:19.285737 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 7951
I20260929 15:52:19.286820 8023 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:52:19.400827 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 7951
2026-09-29T15:52:19Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5195 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:52:19 dist-test-slave-r0ck krb5kdc[8594](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:52:19 dist-test-slave-r0ck krb5kdc[8594](info): set up 2 sockets
Sep 29 15:52:19 dist-test-slave-r0ck krb5kdc[8594](info): commencing operation
krb5kdc: starting...
W20260929 15:52:20.486383 388 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:52:20 dist-test-slave-r0ck krb5kdc[8594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:52:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:52:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:52:20.689564 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46225
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:52:21.001194 8607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:52:21.001531 8607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:52:21.001703 8607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:52:21.009110 8607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:52:21.009240 8607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:52:21.009394 8607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:52:21.009500 8607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:52:21.009572 8607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:52:21.020550 8607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:52:21.021905 8607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:52:21.023177 8607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:52:21.034667 8612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:21.034631 8613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:21.034703 8615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:52:21.035960 8607 server_base.cc:1034] running on GCE node
I20260929 15:52:21.036685 8607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:52:21.038080 8607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:52:21.039312 8607 hybrid_clock.cc:648] HybridClock initialized: now 1790697141039282 us; error 46 us; skew 500 ppm
I20260929 15:52:21.039644 8607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T15:52:21Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1702 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:52:21 dist-test-slave-r0ck krb5kdc[8621](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:52:21 dist-test-slave-r0ck krb5kdc[8621](info): set up 2 sockets
Sep 29 15:52:21 dist-test-slave-r0ck krb5kdc[8621](info): commencing operation
krb5kdc: starting...
W20260929 15:52:22.190279 388 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:52:22 dist-test-slave-r0ck krb5kdc[8621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:52:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:52:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:52:22.400669 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42315
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42315
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:52:22.713624 8634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:52:22.713950 8634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:52:22.714109 8634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:52:22.721331 8634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:52:22.721518 8634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:52:22.721624 8634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:52:22.721721 8634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:52:22.721796 8634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:52:22.731664 8634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:52:22.733038 8634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:52:22.734364 8634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:52:22.746124 8639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:22.746186 8642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:22.746178 8640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:52:22.746968 8634 server_base.cc:1034] running on GCE node
I20260929 15:52:22.747790 8634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:52:22.749181 8634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:52:22.750416 8634 hybrid_clock.cc:648] HybridClock initialized: now 1790697142750371 us; error 61 us; skew 500 ppm
I20260929 15:52:22.750766 8634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:52:22 dist-test-slave-r0ck krb5kdc[8621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:52:22.757875 8634 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:52:22.759465 8634 webserver.cc:460] Webserver started at http://127.0.97.62:33059/ using document root <none> and password file <none>
I20260929 15:52:22.760211 8634 fs_manager.cc:359] Metadata directory not provided
I20260929 15:52:22.760378 8634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:52:22.760715 8634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:52:22.764016 8634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "69e93383f38446f0b4ac3ac57f4993f8"
format_stamp: "Formatted at 2026-09-29 15:52:22 on dist-test-slave-r0ck"
server_key: "8c8cbb84505c7e78ea87b6c9362ba022"
server_key_iv: "6dcdd8b0773855e57b70ea60a821a0b9"
server_key_version: "encryptionkey@0"
I20260929 15:52:22.764878 8634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "69e93383f38446f0b4ac3ac57f4993f8"
format_stamp: "Formatted at 2026-09-29 15:52:22 on dist-test-slave-r0ck"
server_key: "8c8cbb84505c7e78ea87b6c9362ba022"
server_key_iv: "6dcdd8b0773855e57b70ea60a821a0b9"
server_key_version: "encryptionkey@0"
I20260929 15:52:22.770263 8634 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260929 15:52:22.773754 8649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:22.774991 8634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 15:52:22.775182 8634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "69e93383f38446f0b4ac3ac57f4993f8"
format_stamp: "Formatted at 2026-09-29 15:52:22 on dist-test-slave-r0ck"
server_key: "8c8cbb84505c7e78ea87b6c9362ba022"
server_key_iv: "6dcdd8b0773855e57b70ea60a821a0b9"
server_key_version: "encryptionkey@0"
I20260929 15:52:22.775393 8634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:52:22.801231 8634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:52:22.805334 8634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:52:22.805663 8634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:52:22.824774 8634 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42315
I20260929 15:52:22.824800 8700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42315 every 8 connection(s)
I20260929 15:52:22.826170 8634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:52:22.829612 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 8634
I20260929 15:52:22.829805 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:52:22.829993 8701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:52:22.830610 388 external_mini_cluster.cc:1428] Setting key a6a691ae7a765452c0ad9ce31c018a08
I20260929 15:52:22.843190 8701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8: Bootstrap starting.
Sep 29 15:52:22 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:52:22.846766 8701 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8: Neither blocks nor log segments found. Creating new log.
I20260929 15:52:22.847877 8701 log.cc:824] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8: Log is configured to *not* fsync() on all Append() calls
I20260929 15:52:22.850610 8701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8: No bootstrap required, opened a new log
I20260929 15:52:22.854218 8704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:22.832584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55884 (local address 127.0.97.62:42315)
0929 15:52:22.833463 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:22.833475 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:22.833516 (+ 41us) server_negotiation.cc:407] Connection header received
0929 15:52:22.835790 (+ 2274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:22.835813 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:22.836189 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:22.836519 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:22.838231 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:22.839609 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:22.841458 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:22.842164 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:22.842306 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:22.846917 (+ 4611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:22.846927 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:22.846950 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:22.846992 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:22.850230 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:22.850945 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:22.850951 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:22.850960 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:22.851134 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:22.851532 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:22.851537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:22.851539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:22.852027 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:22.852372 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:22.852820 (+ 448us) server_negotiation.cc:299] Negotiation successful
0929 15:52:22.853209 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":413,"threads_started":1}
I20260929 15:52:22.858354 8701 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e93383f38446f0b4ac3ac57f4993f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42315 } }
I20260929 15:52:22.858703 8701 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:52:22.858810 8701 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69e93383f38446f0b4ac3ac57f4993f8, State: Initialized, Role: FOLLOWER
I20260929 15:52:22.859382 8701 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e93383f38446f0b4ac3ac57f4993f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42315 } }
I20260929 15:52:22.859615 8701 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:52:22.859751 8701 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:52:22.859918 8701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:52:22.862130 8701 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e93383f38446f0b4ac3ac57f4993f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42315 } }
I20260929 15:52:22.862591 8701 leader_election.cc:302] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69e93383f38446f0b4ac3ac57f4993f8; no voters:
I20260929 15:52:22.863142 8701 leader_election.cc:288] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:52:22.863320 8706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:52:22.864642 8706 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8 [term 1 LEADER]: Becoming Leader. State: Replica: 69e93383f38446f0b4ac3ac57f4993f8, State: Running, Role: LEADER
I20260929 15:52:22.865185 8701 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:52:22.865206 8706 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e93383f38446f0b4ac3ac57f4993f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42315 } }
I20260929 15:52:22.868446 8707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69e93383f38446f0b4ac3ac57f4993f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e93383f38446f0b4ac3ac57f4993f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42315 } } }
I20260929 15:52:22.868929 8707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8 [sys.catalog]: This master's current role is: LEADER
I20260929 15:52:22.868418 8708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69e93383f38446f0b4ac3ac57f4993f8. Latest consensus state: current_term: 1 leader_uuid: "69e93383f38446f0b4ac3ac57f4993f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e93383f38446f0b4ac3ac57f4993f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42315 } } }
I20260929 15:52:22.869510 8708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8 [sys.catalog]: This master's current role is: LEADER
I20260929 15:52:22.870970 8713 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:52:22.876736 8713 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:52:22.886058 8713 catalog_manager.cc:1347] Generated new cluster ID: 27c3832082c34e1eb691468225c731ed
I20260929 15:52:22.886181 8713 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:52:22.896955 8713 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:52:22.898262 8713 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:52:22.910430 8713 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8: Generated new TSK 0
I20260929 15:52:22.911585 8713 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 15:52:22.969236 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42315
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:52:23.290113 8727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:52:23.290369 8727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:52:23.290460 8727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:52:23.297461 8727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:52:23.297554 8727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:52:23.297660 8727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:52:23.306861 8727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:52:23.308058 8727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:52:23.309240 8727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:52:23.321666 8733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:23.321758 8732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:23.321732 8735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:52:23.322914 8727 server_base.cc:1034] running on GCE node
I20260929 15:52:23.323475 8727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:52:23.324352 8727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:52:23.325537 8727 hybrid_clock.cc:648] HybridClock initialized: now 1790697143325495 us; error 36 us; skew 500 ppm
I20260929 15:52:23.325829 8727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:52:23 dist-test-slave-r0ck krb5kdc[8621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:52:23.328814 8727 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:52:23.330216 8727 webserver.cc:460] Webserver started at http://127.0.97.1:38275/ using document root <none> and password file <none>
I20260929 15:52:23.330859 8727 fs_manager.cc:359] Metadata directory not provided
I20260929 15:52:23.330982 8727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:52:23.331290 8727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:52:23.334211 8727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-0/data/instance:
uuid: "e831770c7475410e9152a2b865bbbb0a"
format_stamp: "Formatted at 2026-09-29 15:52:23 on dist-test-slave-r0ck"
server_key: "f7815ba80bec31b352bf1a741c1fe146"
server_key_iv: "c66e5947296dbdf1fdd412328419f899"
server_key_version: "encryptionkey@0"
I20260929 15:52:23.335227 8727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e831770c7475410e9152a2b865bbbb0a"
format_stamp: "Formatted at 2026-09-29 15:52:23 on dist-test-slave-r0ck"
server_key: "f7815ba80bec31b352bf1a741c1fe146"
server_key_iv: "c66e5947296dbdf1fdd412328419f899"
server_key_version: "encryptionkey@0"
I20260929 15:52:23.340266 8727 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 15:52:23.344094 8742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:23.345611 8727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 15:52:23.345830 8727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-0/wal
uuid: "e831770c7475410e9152a2b865bbbb0a"
format_stamp: "Formatted at 2026-09-29 15:52:23 on dist-test-slave-r0ck"
server_key: "f7815ba80bec31b352bf1a741c1fe146"
server_key_iv: "c66e5947296dbdf1fdd412328419f899"
server_key_version: "encryptionkey@0"
I20260929 15:52:23.346077 8727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:52:23.391767 8727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:52:23.395809 8727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:52:23.396121 8727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:52:23.397042 8727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:52:23.398417 8727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:52:23.398516 8727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:23.398660 8727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:52:23.398734 8727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:23.431242 8727 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38521
I20260929 15:52:23.431592 8854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38521 every 8 connection(s)
I20260929 15:52:23.432858 8727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:52:23.439961 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 8727
I20260929 15:52:23.440177 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:52:23.440778 388 external_mini_cluster.cc:1428] Setting key ddab718221c61b997895305e3635cb6c
Sep 29 15:52:23 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:52:23.452742 8857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:23.435268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35707 (local address 127.0.97.62:42315)
0929 15:52:23.435710 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:23.435715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:23.436528 (+ 813us) server_negotiation.cc:407] Connection header received
0929 15:52:23.438430 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:23.438436 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:23.438532 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:23.438671 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:23.440091 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:23.440861 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:23.442677 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:23.443068 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:23.443202 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:23.446537 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:23.446541 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:23.446545 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:23.446574 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:23.449782 (+ 3208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:23.450563 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:23.450568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:23.450572 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:23.450691 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:23.451103 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:23.451108 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:23.451112 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:23.451380 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:23.451526 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:23.452176 (+ 650us) server_negotiation.cc:299] Negotiation successful
0929 15:52:23.452396 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":247,"threads_started":1}
I20260929 15:52:23.453723 8858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:23.435652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.1:35707)
0929 15:52:23.436351 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:23.436387 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:23.438158 (+ 1771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:23.438904 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:23.438912 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:23.439374 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:23.439924 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:23.439942 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:23.441040 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:23.441044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:23.442544 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:23.442556 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:23.443282 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:23.443290 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:23.443524 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:23.444190 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:23.444211 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:23.446318 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:23.449950 (+ 3632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:23.449957 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:23.449973 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:23.450441 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:23.450795 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:23.450801 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:23.450805 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:23.450989 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:23.451501 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:23.451509 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:23.451821 (+ 312us) client_negotiation.cc:769] Sending connection context
0929 15:52:23.452069 (+ 248us) client_negotiation.cc:240] Negotiation successful
0929 15:52:23.452569 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":301,"threads_started":1}
I20260929 15:52:23.455261 8855 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42315
I20260929 15:52:23.455542 8855 heartbeater.cc:464] Registering TS with master...
I20260929 15:52:23.456202 8855 heartbeater.cc:511] Master 127.0.97.62:42315 requested a full tablet report, sending...
I20260929 15:52:23.458158 8666 ts_manager.cc:194] Registered new tserver with Master: e831770c7475410e9152a2b865bbbb0a (127.0.97.1:38521)
I20260929 15:52:23.459556 8666 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:35707
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 15:52:23.489560 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42315
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:52:23.802148 8861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:52:23.802515 8861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:52:23.802700 8861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:52:23.812958 8861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:52:23.813064 8861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:52:23.813175 8861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:52:23.823413 8861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:52:23.824606 8861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:52:23.825937 8861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:52:23.840688 8866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:23.840736 8869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:23.840688 8867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:52:23.841516 8861 server_base.cc:1034] running on GCE node
I20260929 15:52:23.842116 8861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:52:23.843022 8861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:52:23.844250 8861 hybrid_clock.cc:648] HybridClock initialized: now 1790697143844200 us; error 67 us; skew 500 ppm
I20260929 15:52:23.844558 8861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:52:23 dist-test-slave-r0ck krb5kdc[8621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:52:23.847719 8861 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:52:23.849057 8861 webserver.cc:460] Webserver started at http://127.0.97.2:42113/ using document root <none> and password file <none>
I20260929 15:52:23.849781 8861 fs_manager.cc:359] Metadata directory not provided
I20260929 15:52:23.849926 8861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:52:23.850265 8861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:52:23.853060 8861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-1/data/instance:
uuid: "8c9355a73610483cb67e1977e7e33831"
format_stamp: "Formatted at 2026-09-29 15:52:23 on dist-test-slave-r0ck"
server_key: "c681ecec8bf6ba67379a4ca18bfa5f8d"
server_key_iv: "5d58c5ed2ca3f3efe7196a092766302c"
server_key_version: "encryptionkey@0"
I20260929 15:52:23.853813 8861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8c9355a73610483cb67e1977e7e33831"
format_stamp: "Formatted at 2026-09-29 15:52:23 on dist-test-slave-r0ck"
server_key: "c681ecec8bf6ba67379a4ca18bfa5f8d"
server_key_iv: "5d58c5ed2ca3f3efe7196a092766302c"
server_key_version: "encryptionkey@0"
I20260929 15:52:23.858659 8861 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 15:52:23.861987 8876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:23.863324 8861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:52:23.863497 8861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-1/wal
uuid: "8c9355a73610483cb67e1977e7e33831"
format_stamp: "Formatted at 2026-09-29 15:52:23 on dist-test-slave-r0ck"
server_key: "c681ecec8bf6ba67379a4ca18bfa5f8d"
server_key_iv: "5d58c5ed2ca3f3efe7196a092766302c"
server_key_version: "encryptionkey@0"
I20260929 15:52:23.863689 8861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:52:23.892522 8861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:52:23.896497 8861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:52:23.896785 8861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:52:23.897737 8861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:52:23.898983 8861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:52:23.899083 8861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:23.899230 8861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:52:23.899302 8861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:23.931847 8861 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36825
I20260929 15:52:23.931942 8988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36825 every 8 connection(s)
I20260929 15:52:23.933099 8861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:52:23.939393 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 8861
I20260929 15:52:23.939584 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:52:23.940202 388 external_mini_cluster.cc:1428] Setting key ecabc6c6a1dc904d1db0668ba1d075a7
Sep 29 15:52:23 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:52:23.954567 8857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:23.935571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59869 (local address 127.0.97.62:42315)
0929 15:52:23.935779 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:23.935785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:23.936912 (+ 1127us) server_negotiation.cc:407] Connection header received
0929 15:52:23.938721 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:23.938725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:23.938789 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:23.938877 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:23.941004 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:23.942187 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:23.944059 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:23.944509 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:23.944659 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:23.948329 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:23.948336 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:23.948344 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:23.948388 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:23.951236 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:23.952200 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:23.952203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:23.952206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:23.952317 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:23.952806 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:23.952809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:23.952811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:23.953044 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:23.953146 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:23.954055 (+ 909us) server_negotiation.cc:299] Negotiation successful
0929 15:52:23.954304 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 15:52:23.956136 8991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:23.936078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.2:59869)
0929 15:52:23.936755 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:23.936792 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:23.938493 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:23.939210 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:23.939241 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:23.940007 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:23.940820 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:23.940841 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:23.942403 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:23.942411 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:23.943895 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:23.943908 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:23.944766 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:23.944774 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:23.945005 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:23.945786 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:23.945810 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:23.948069 (+ 2259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:23.951370 (+ 3301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:23.951382 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:23.951400 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:23.952056 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:23.952430 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:23.952439 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:23.952446 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:23.952696 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:23.953177 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:23.953189 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:23.953672 (+ 483us) client_negotiation.cc:769] Sending connection context
0929 15:52:23.953998 (+ 326us) client_negotiation.cc:240] Negotiation successful
0929 15:52:23.954598 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":279,"threads_started":1}
I20260929 15:52:23.957630 8989 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42315
I20260929 15:52:23.957991 8989 heartbeater.cc:464] Registering TS with master...
I20260929 15:52:23.958906 8989 heartbeater.cc:511] Master 127.0.97.62:42315 requested a full tablet report, sending...
I20260929 15:52:23.960693 8666 ts_manager.cc:194] Registered new tserver with Master: 8c9355a73610483cb67e1977e7e33831 (127.0.97.2:36825)
I20260929 15:52:23.962122 8666 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59869
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 15:52:23.992974 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42315
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:52:24.309875 8994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:52:24.310206 8994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:52:24.310343 8994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:52:24.317539 8994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:52:24.317646 8994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:52:24.317776 8994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:52:24.327483 8994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:52:24.328603 8994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:52:24.329859 8994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:52:24.342466 8999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:24.342561 9000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:24.342592 9002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:52:24.343695 8994 server_base.cc:1034] running on GCE node
I20260929 15:52:24.344252 8994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:52:24.345130 8994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:52:24.346346 8994 hybrid_clock.cc:648] HybridClock initialized: now 1790697144346289 us; error 49 us; skew 500 ppm
I20260929 15:52:24.346638 8994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:52:24 dist-test-slave-r0ck krb5kdc[8621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:52:24.349707 8994 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:52:24.351112 8994 webserver.cc:460] Webserver started at http://127.0.97.3:46735/ using document root <none> and password file <none>
I20260929 15:52:24.351780 8994 fs_manager.cc:359] Metadata directory not provided
I20260929 15:52:24.351920 8994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:52:24.352210 8994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:52:24.354964 8994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-2/data/instance:
uuid: "6633d9a21a0343368695ad7adb972879"
format_stamp: "Formatted at 2026-09-29 15:52:24 on dist-test-slave-r0ck"
server_key: "2580e62d612aa405ed15eb8ff98f0870"
server_key_iv: "d1375c96bfc5805de0eb8c6214cf5bad"
server_key_version: "encryptionkey@0"
I20260929 15:52:24.355708 8994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6633d9a21a0343368695ad7adb972879"
format_stamp: "Formatted at 2026-09-29 15:52:24 on dist-test-slave-r0ck"
server_key: "2580e62d612aa405ed15eb8ff98f0870"
server_key_iv: "d1375c96bfc5805de0eb8c6214cf5bad"
server_key_version: "encryptionkey@0"
I20260929 15:52:24.360339 8994 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 15:52:24.363592 9009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:24.364806 8994 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 15:52:24.364974 8994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-2/wal
uuid: "6633d9a21a0343368695ad7adb972879"
format_stamp: "Formatted at 2026-09-29 15:52:24 on dist-test-slave-r0ck"
server_key: "2580e62d612aa405ed15eb8ff98f0870"
server_key_iv: "d1375c96bfc5805de0eb8c6214cf5bad"
server_key_version: "encryptionkey@0"
I20260929 15:52:24.365159 8994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:52:24.396555 8994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:52:24.401005 8994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:52:24.401304 8994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:52:24.402456 8994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:52:24.403824 8994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:52:24.403934 8994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:24.404093 8994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:52:24.404166 8994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:24.439198 8994 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39741
I20260929 15:52:24.439254 9121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39741 every 8 connection(s)
I20260929 15:52:24.440361 8994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:52:24.443645 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 8994
I20260929 15:52:24.443840 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:52:24.444346 388 external_mini_cluster.cc:1428] Setting key 0faacc074b008e2fc73fc1a5d3a5225a
Sep 29 15:52:24 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:52:24.459556 8857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:24.442934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44175 (local address 127.0.97.62:42315)
0929 15:52:24.443172 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:24.443177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:24.444184 (+ 1007us) server_negotiation.cc:407] Connection header received
0929 15:52:24.446395 (+ 2211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:24.446399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:24.446469 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:24.446547 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:24.448113 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:24.448878 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:24.450474 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:24.450859 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:24.451018 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:24.454130 (+ 3112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:24.454134 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:24.454137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:24.454176 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:24.456482 (+ 2306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:24.457135 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:24.457139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:24.457141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:24.457249 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:24.457709 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:24.457712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:24.457714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:24.458062 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:24.458197 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:24.459003 (+ 806us) server_negotiation.cc:299] Negotiation successful
0929 15:52:24.459207 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 15:52:24.460453 9124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:24.443176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.3:44175)
0929 15:52:24.443979 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:24.444032 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:24.446110 (+ 2078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:24.446807 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:24.446818 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:24.447430 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:24.447957 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:24.447972 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:24.449003 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:24.449007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:24.450328 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:24.450340 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:24.451079 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:24.451084 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:24.451261 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:24.451951 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:24.451972 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:24.453964 (+ 1992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:24.456625 (+ 2661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:24.456630 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:24.456642 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:24.457021 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:24.457388 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:24.457399 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:24.457402 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:24.457564 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:24.458186 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:24.458195 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:24.458595 (+ 400us) client_negotiation.cc:769] Sending connection context
0929 15:52:24.458888 (+ 293us) client_negotiation.cc:240] Negotiation successful
0929 15:52:24.459403 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":356,"threads_started":1}
I20260929 15:52:24.461778 9122 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42315
I20260929 15:52:24.462067 9122 heartbeater.cc:464] Registering TS with master...
I20260929 15:52:24.462445 8855 heartbeater.cc:503] Master 127.0.97.62:42315 was elected leader, sending a full tablet report...
I20260929 15:52:24.462857 9122 heartbeater.cc:511] Master 127.0.97.62:42315 requested a full tablet report, sending...
I20260929 15:52:24.464303 8666 ts_manager.cc:194] Registered new tserver with Master: 6633d9a21a0343368695ad7adb972879 (127.0.97.3:39741)
I20260929 15:52:24.465022 8666 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44175
I20260929 15:52:24.471479 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260929 15:52:24.604483 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 8727
Sep 29 15:52:24 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 15:52:24.620720 9132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:24.605675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53428 (local address 127.0.97.1:38521)
0929 15:52:24.606323 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:24.606331 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:24.606354 (+ 23us) server_negotiation.cc:407] Connection header received
0929 15:52:24.606409 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:24.606416 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:24.606623 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:24.606788 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:24.608073 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:24.609068 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:24.610347 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:24.610885 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:24.611028 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:24.614883 (+ 3855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:24.614893 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:24.614906 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:24.614947 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:24.618130 (+ 3183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:24.618827 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:24.618835 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:24.618837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:24.618957 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:24.619278 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:24.619282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:24.619284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:24.619598 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:24.619824 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:24.620141 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 15:52:24.620383 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":315,"threads_started":1}
I20260929 15:52:24.622035 8850 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:52:24.703279 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 8727
I20260929 15:52:24.719256 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 8861
Sep 29 15:52:24 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 15:52:24.734498 9134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:24.720156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52304 (local address 127.0.97.2:36825)
0929 15:52:24.720610 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:24.720617 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:24.720643 (+ 26us) server_negotiation.cc:407] Connection header received
0929 15:52:24.720701 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:24.720707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:24.720918 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:24.721043 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:24.722578 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:24.723540 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:24.724711 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:24.725260 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:24.725436 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:24.729594 (+ 4158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:24.729605 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:24.729618 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:24.729654 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:24.732111 (+ 2457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:24.732708 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:24.732715 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:24.732717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:24.732826 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:24.733118 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:24.733121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:24.733123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:24.733481 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:24.733694 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:24.733926 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 15:52:24.734106 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":241,"threads_started":1}
I20260929 15:52:24.735692 8984 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:52:24.816188 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 8861
I20260929 15:52:24.831635 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 8994
Sep 29 15:52:24 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 15:52:24.847280 9136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:24.832540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36880 (local address 127.0.97.3:39741)
0929 15:52:24.833113 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:24.833120 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:24.833147 (+ 27us) server_negotiation.cc:407] Connection header received
0929 15:52:24.833205 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:24.833213 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:24.833485 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:24.833612 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:24.834845 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:24.835852 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:24.837042 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:24.837705 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:24.837806 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:24.841981 (+ 4175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:24.841991 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:24.842004 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:24.842033 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:24.844740 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:24.845321 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:24.845327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:24.845329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:24.845548 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:24.845932 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:24.845935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:24.845937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:24.846240 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:24.846454 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:24.846738 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 15:52:24.846947 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":305,"threads_started":1}
I20260929 15:52:24.848493 9117 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:52:24.928819 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 8994
I20260929 15:52:24.944195 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 8634
I20260929 15:52:24.945457 8696 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:52:25.045274 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 8634
I20260929 15:52:25.065310 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42315
--webserver_interface=127.0.97.62
--webserver_port=33059
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42315
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:52:25.372866 9139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:52:25.373178 9139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:52:25.373351 9139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:52:25.380291 9139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:52:25.380445 9139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:52:25.380551 9139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:52:25.380654 9139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:52:25.380744 9139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:52:25.390955 9139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=33059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:52:25.392295 9139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:52:25.393643 9139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:52:25.405099 9144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:25.405045 9145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:25.405017 9147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:52:25.405923 9139 server_base.cc:1034] running on GCE node
I20260929 15:52:25.406628 9139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:52:25.408197 9139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:52:25.409452 9139 hybrid_clock.cc:648] HybridClock initialized: now 1790697145409404 us; error 62 us; skew 500 ppm
I20260929 15:52:25.409796 9139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:52:25 dist-test-slave-r0ck krb5kdc[8621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:52:25.412720 9139 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:52:25.414223 9139 webserver.cc:460] Webserver started at http://127.0.97.62:33059/ using document root <none> and password file <none>
I20260929 15:52:25.414891 9139 fs_manager.cc:359] Metadata directory not provided
I20260929 15:52:25.415047 9139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:52:25.420475 9139 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 15:52:25.423324 9154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:25.424778 9139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 15:52:25.424966 9139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "69e93383f38446f0b4ac3ac57f4993f8"
format_stamp: "Formatted at 2026-09-29 15:52:22 on dist-test-slave-r0ck"
server_key: "8c8cbb84505c7e78ea87b6c9362ba022"
server_key_iv: "6dcdd8b0773855e57b70ea60a821a0b9"
server_key_version: "encryptionkey@0"
I20260929 15:52:25.425984 9139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:52:25.441843 9139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:52:25.446375 9139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:52:25.446642 9139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:52:25.464949 9139 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42315
I20260929 15:52:25.464982 9205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42315 every 8 connection(s)
I20260929 15:52:25.466353 9139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:52:25.472857 9206 sys_catalog.cc:263] Verifying existing consensus state
I20260929 15:52:25.474629 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 9139
I20260929 15:52:25.475535 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:38521
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:42315
--webserver_port=38275
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:52:25.476862 9206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8: Bootstrap starting.
I20260929 15:52:25.497887 9206 log.cc:824] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8: Log is configured to *not* fsync() on all Append() calls
I20260929 15:52:25.506181 9206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 15:52:25.506695 9206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8: Bootstrap complete.
I20260929 15:52:25.515540 9206 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e93383f38446f0b4ac3ac57f4993f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42315 } }
I20260929 15:52:25.516297 9206 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69e93383f38446f0b4ac3ac57f4993f8, State: Initialized, Role: FOLLOWER
I20260929 15:52:25.516958 9206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e93383f38446f0b4ac3ac57f4993f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42315 } }
I20260929 15:52:25.517189 9206 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:52:25.517320 9206 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:52:25.517539 9206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8 [term 1 FOLLOWER]: Advancing to term 2
I20260929 15:52:25.521044 9206 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e93383f38446f0b4ac3ac57f4993f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42315 } }
I20260929 15:52:25.521622 9206 leader_election.cc:302] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69e93383f38446f0b4ac3ac57f4993f8; no voters:
I20260929 15:52:25.522508 9206 leader_election.cc:288] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 15:52:25.522921 9210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 15:52:25.525964 9210 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8 [term 2 LEADER]: Becoming Leader. State: Replica: 69e93383f38446f0b4ac3ac57f4993f8, State: Running, Role: LEADER
I20260929 15:52:25.526600 9210 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e93383f38446f0b4ac3ac57f4993f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42315 } }
I20260929 15:52:25.526947 9206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:52:25.529163 9212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69e93383f38446f0b4ac3ac57f4993f8. Latest consensus state: current_term: 2 leader_uuid: "69e93383f38446f0b4ac3ac57f4993f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e93383f38446f0b4ac3ac57f4993f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42315 } } }
I20260929 15:52:25.529558 9212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8 [sys.catalog]: This master's current role is: LEADER
I20260929 15:52:25.529275 9211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "69e93383f38446f0b4ac3ac57f4993f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e93383f38446f0b4ac3ac57f4993f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42315 } } }
I20260929 15:52:25.529753 9211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8 [sys.catalog]: This master's current role is: LEADER
I20260929 15:52:25.540903 9215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:52:25.546350 9215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:52:25.547724 9215 catalog_manager.cc:1259] Loaded cluster ID: 27c3832082c34e1eb691468225c731ed
I20260929 15:52:25.547812 9215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:52:25.551703 9215 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:52:25.553539 9215 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 69e93383f38446f0b4ac3ac57f4993f8: Loaded TSK: 0
I20260929 15:52:25.554893 9215 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 15:52:25.810105 9208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:52:25.810401 9208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:52:25.810562 9208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:52:25.817775 9208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:52:25.817930 9208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:52:25.818107 9208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:52:25.827896 9208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:38521
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=38275
--tserver_master_addrs=127.0.97.62:42315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:52:25.829255 9208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:52:25.830592 9208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:52:25.845175 9232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:25.845239 9233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:25.845218 9235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:52:25.845714 9208 server_base.cc:1034] running on GCE node
I20260929 15:52:25.846467 9208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:52:25.847467 9208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:52:25.848664 9208 hybrid_clock.cc:648] HybridClock initialized: now 1790697145848611 us; error 49 us; skew 500 ppm
I20260929 15:52:25.848997 9208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:52:25 dist-test-slave-r0ck krb5kdc[8621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697145, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:52:25.852027 9208 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260929 15:52:25.853545 9208 webserver.cc:460] Webserver started at http://127.0.97.1:38275/ using document root <none> and password file <none>
I20260929 15:52:25.854188 9208 fs_manager.cc:359] Metadata directory not provided
I20260929 15:52:25.854321 9208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:52:25.859719 9208 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 15:52:25.863215 9242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:25.865037 9208 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 15:52:25.865199 9208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-0/wal
uuid: "e831770c7475410e9152a2b865bbbb0a"
format_stamp: "Formatted at 2026-09-29 15:52:23 on dist-test-slave-r0ck"
server_key: "f7815ba80bec31b352bf1a741c1fe146"
server_key_iv: "c66e5947296dbdf1fdd412328419f899"
server_key_version: "encryptionkey@0"
I20260929 15:52:25.866573 9208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:52:25.882602 9208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:52:25.886874 9208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:52:25.887166 9208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:52:25.888223 9208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:52:25.889652 9208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:52:25.889739 9208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:25.889881 9208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:52:25.889985 9208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:25.923308 9208 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38521
I20260929 15:52:25.923399 9354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38521 every 8 connection(s)
I20260929 15:52:25.924690 9208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:52:25.928534 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 9208
I20260929 15:52:25.929263 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:36825
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:42315
--webserver_port=42113
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 15:52:25 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:25 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:25.944300 9358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:25.927813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.1:56243)
0929 15:52:25.928736 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:25.928782 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:25.931722 (+ 2940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:25.933061 (+ 1339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:25.933074 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:25.933841 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:25.934571 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:25.934598 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:25.936294 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:25.936298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:25.937607 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:25.937621 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:25.938612 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:25.938619 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:25.938902 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:25.939593 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:25.939614 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:25.943101 (+ 3487us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":339,"threads_started":1}
W20260929 15:52:25.944377 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:25.927728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56243 (local address 127.0.97.62:42315)
0929 15:52:25.929797 (+ 2069us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:25.929807 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:25.929876 (+ 69us) server_negotiation.cc:407] Connection header received
0929 15:52:25.932248 (+ 2372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:25.932260 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:25.932580 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:25.932859 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:25.934770 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:25.936138 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:25.937756 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:25.938386 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:25.938581 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:25.943463 (+ 4882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:56243: BlockingRecv error: recv got EOF from 127.0.97.1:56243 (error 108)
Metrics: {"server-negotiator.queue_time_us":1826,"thread_start_us":1612,"threads_started":1}
W20260929 15:52:25.945144 9355 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42315 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:42315: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 29 15:52:25 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:25 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:25.954267 9358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:25.945787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.1:58331)
0929 15:52:25.945961 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:25.945979 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:25.946081 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:25.946467 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:25.946470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:25.946734 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:25.946858 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:25.946865 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:25.948405 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:25.948409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:25.949307 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:25.949330 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:25.950463 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:25.950468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:25.950601 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:25.951198 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:25.951214 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:25.954004 (+ 2790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 15:52:25.954463 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:25.945961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58331 (local address 127.0.97.62:42315)
0929 15:52:25.946162 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:25.946169 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:25.946190 (+ 21us) server_negotiation.cc:407] Connection header received
0929 15:52:25.946248 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:25.946254 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:25.946339 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:25.946462 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:25.946993 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:25.948264 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:25.949506 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:25.950264 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:25.950412 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:25.954244 (+ 3832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:58331: BlockingRecv error: recv got EOF from 127.0.97.1:58331 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 15:52:25 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:25 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:25.963413 9358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:25.955096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.1:52057)
0929 15:52:25.955267 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:25.955286 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:25.955377 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:25.955902 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:25.955909 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:25.956316 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:25.956541 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:25.956556 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:25.958014 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:25.958018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:25.959036 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:25.959047 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:25.959927 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:25.959931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:25.960043 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:25.960609 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:25.960625 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:25.963209 (+ 2584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 15:52:25.963688 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:25.955178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52057 (local address 127.0.97.62:42315)
0929 15:52:25.955324 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:25.955328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:25.955343 (+ 15us) server_negotiation.cc:407] Connection header received
0929 15:52:25.955488 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:25.955492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:25.955576 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:25.955675 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:25.956706 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:25.957865 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:25.959173 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:25.959798 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:25.959915 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:25.963432 (+ 3517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:52057: BlockingRecv error: recv got EOF from 127.0.97.1:52057 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260929 15:52:25.963833 9355 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 15:52:26.253598 9359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:52:26.253858 9359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:52:26.254016 9359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:52:26.261682 9359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:52:26.261782 9359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:52:26.261972 9359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:52:26.271449 9359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:36825
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=42113
--tserver_master_addrs=127.0.97.62:42315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:52:26.272840 9359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:52:26.274060 9359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:52:26.286257 9364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:26.286337 9367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:26.286305 9365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:52:26.287178 9359 server_base.cc:1034] running on GCE node
I20260929 15:52:26.287678 9359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:52:26.288486 9359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:52:26.289683 9359 hybrid_clock.cc:648] HybridClock initialized: now 1790697146289649 us; error 32 us; skew 500 ppm
I20260929 15:52:26.290015 9359 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:52:26 dist-test-slave-r0ck krb5kdc[8621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697146, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:52:26.292925 9359 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260929 15:52:26.294219 9359 webserver.cc:460] Webserver started at http://127.0.97.2:42113/ using document root <none> and password file <none>
I20260929 15:52:26.294811 9359 fs_manager.cc:359] Metadata directory not provided
I20260929 15:52:26.294912 9359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:52:26.299747 9359 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 15:52:26.302207 9374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:26.303459 9359 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:52:26.303644 9359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-1/wal
uuid: "8c9355a73610483cb67e1977e7e33831"
format_stamp: "Formatted at 2026-09-29 15:52:23 on dist-test-slave-r0ck"
server_key: "c681ecec8bf6ba67379a4ca18bfa5f8d"
server_key_iv: "5d58c5ed2ca3f3efe7196a092766302c"
server_key_version: "encryptionkey@0"
I20260929 15:52:26.304565 9359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:52:26.324301 9359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:52:26.328289 9359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:52:26.328536 9359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:52:26.329442 9359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:52:26.330767 9359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:52:26.330890 9359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:26.331015 9359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:52:26.331104 9359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:26.362360 9359 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36825
I20260929 15:52:26.362438 9486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36825 every 8 connection(s)
I20260929 15:52:26.363686 9359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:52:26.371074 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 9359
I20260929 15:52:26.371769 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:39741
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:42315
--webserver_port=46735
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 15:52:26 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:26 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 15:52:26.380643 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:26.366129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56907 (local address 127.0.97.62:42315)
0929 15:52:26.366325 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:26.366331 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:26.367530 (+ 1199us) server_negotiation.cc:407] Connection header received
0929 15:52:26.369267 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:26.369271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:26.369333 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:26.369486 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:26.370869 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:26.371944 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:26.374478 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:26.374868 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:26.374970 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:26.380403 (+ 5433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:56907: BlockingRecv error: recv got EOF from 127.0.97.2:56907 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 15:52:26.381480 9489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:26.366659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.2:56907)
0929 15:52:26.367357 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:26.367393 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:26.369031 (+ 1638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:26.369674 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:26.369682 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:26.370165 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:26.370698 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:26.370714 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:26.372136 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:26.372144 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:26.374339 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:26.374356 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:26.375095 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:26.375105 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:26.375424 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:26.376151 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:26.376189 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:26.380242 (+ 4053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":281,"threads_started":1}
W20260929 15:52:26.382121 9487 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42315 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:42315: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 29 15:52:26 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:26 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:26.390741 9489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:26.382686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.2:37737)
0929 15:52:26.382844 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:26.382861 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:26.382951 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:26.383255 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:26.383258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:26.383526 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:26.383648 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:26.383654 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:26.385043 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:26.385046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:26.385964 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:26.385975 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:26.386965 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:26.386969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:26.387081 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:26.387638 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:26.387654 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:26.390550 (+ 2896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 15:52:26.391144 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:26.382779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37737 (local address 127.0.97.62:42315)
0929 15:52:26.382958 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:26.382966 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:26.382988 (+ 22us) server_negotiation.cc:407] Connection header received
0929 15:52:26.383040 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:26.383048 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:26.383130 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:26.383242 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:26.383776 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:26.384919 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:26.386094 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:26.386817 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:26.386968 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:26.390855 (+ 3887us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:37737: BlockingRecv error: recv got EOF from 127.0.97.2:37737 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 15:52:26 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:26 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:26.398737 9489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:26.391477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.2:43003)
0929 15:52:26.391632 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:26.391651 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:26.391769 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:26.392063 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:26.392067 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:26.392352 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:26.392489 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:26.392497 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:26.393450 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:26.393458 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:26.394348 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:26.394358 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:26.395026 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:26.395031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:26.395200 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:26.395793 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:26.395809 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:26.398452 (+ 2643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 15:52:26.399051 9487 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 15:52:26.399224 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:26.391623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43003 (local address 127.0.97.62:42315)
0929 15:52:26.391797 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:26.391805 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:26.391825 (+ 20us) server_negotiation.cc:407] Connection header received
0929 15:52:26.391875 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:26.391879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:26.391943 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:26.392042 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:26.392619 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:26.393284 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:26.394488 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:26.394891 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:26.395018 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:26.398645 (+ 3627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:43003: BlockingRecv error: recv got EOF from 127.0.97.2:43003 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260929 15:52:26.693579 9490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:52:26.693830 9490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:52:26.693922 9490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:52:26.701015 9490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:52:26.701134 9490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:52:26.701246 9490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:52:26.711058 9490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:39741
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=46735
--tserver_master_addrs=127.0.97.62:42315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:52:26.712205 9490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:52:26.713449 9490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:52:26.725790 9496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:26.725873 9495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:26.725905 9498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:52:26.726441 9490 server_base.cc:1034] running on GCE node
I20260929 15:52:26.727037 9490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:52:26.727924 9490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:52:26.729161 9490 hybrid_clock.cc:648] HybridClock initialized: now 1790697146729101 us; error 47 us; skew 500 ppm
I20260929 15:52:26.729517 9490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:52:26 dist-test-slave-r0ck krb5kdc[8621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697146, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:52:26.732474 9490 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260929 15:52:26.733762 9490 webserver.cc:460] Webserver started at http://127.0.97.3:46735/ using document root <none> and password file <none>
I20260929 15:52:26.734338 9490 fs_manager.cc:359] Metadata directory not provided
I20260929 15:52:26.734437 9490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:52:26.739192 9490 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 15:52:26.741541 9505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:26.742676 9490 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 15:52:26.742821 9490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-2/wal
uuid: "6633d9a21a0343368695ad7adb972879"
format_stamp: "Formatted at 2026-09-29 15:52:24 on dist-test-slave-r0ck"
server_key: "2580e62d612aa405ed15eb8ff98f0870"
server_key_iv: "d1375c96bfc5805de0eb8c6214cf5bad"
server_key_version: "encryptionkey@0"
I20260929 15:52:26.743640 9490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:52:26.756737 9490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:52:26.760672 9490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:52:26.760883 9490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:52:26.761744 9490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:52:26.762885 9490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:52:26.762972 9490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:26.763044 9490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:52:26.763098 9490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:26.794350 9490 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39741
I20260929 15:52:26.794435 9617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39741 every 8 connection(s)
I20260929 15:52:26.795707 9490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:52:26.796408 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 9490
Sep 29 15:52:26 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:26 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 15:52:26.811865 9621 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:26.798915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59965 (local address 127.0.97.62:42315)
0929 15:52:26.799338 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:26.799342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:26.800373 (+ 1031us) server_negotiation.cc:407] Connection header received
0929 15:52:26.802520 (+ 2147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:26.802525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:26.802596 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:26.802713 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:26.804048 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:26.804764 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:26.806276 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:26.806690 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:26.806822 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:26.811553 (+ 4731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:59965: BlockingRecv error: recv got EOF from 127.0.97.3:59965 (error 108)
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":246,"threads_started":1}
I20260929 15:52:26.811897 9357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:26.797425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55898 (local address 127.0.97.62:42315)
0929 15:52:26.797604 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:26.797610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:26.798043 (+ 433us) server_negotiation.cc:407] Connection header received
0929 15:52:26.798158 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:26.798164 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:26.798247 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:26.798353 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:26.800246 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:26.801298 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:26.802928 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:26.803358 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:26.803486 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:26.805947 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:26.805954 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:26.805966 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:26.805998 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:26.808919 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:26.809695 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:26.809701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:26.809706 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:26.809864 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:26.810216 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:26.810225 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:26.810227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:26.810598 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:26.810915 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:26.811252 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 15:52:26.811595 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 15:52:26.812438 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:26.799463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.3:59965)
0929 15:52:26.800199 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:26.800258 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:26.802296 (+ 2038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:26.802946 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:26.802953 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:26.803399 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:26.803890 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:26.803905 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:26.804914 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:26.804918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:26.806130 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:26.806141 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:26.806906 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:26.806915 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:26.807138 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:26.807915 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:26.807951 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:26.811333 (+ 3382us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":298,"threads_started":1}
W20260929 15:52:26.813076 9618 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42315 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:42315: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 29 15:52:26 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:26 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:26.820688 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:26.813652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.3:51355)
0929 15:52:26.813857 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:26.813876 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:26.813966 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:26.814268 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:26.814273 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:26.814642 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:26.814830 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:26.814844 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:26.815907 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:26.815911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:26.816766 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:26.816776 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:26.817409 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:26.817414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:26.817541 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:26.818027 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:26.818042 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:26.820508 (+ 2466us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117}
W20260929 15:52:26.820917 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:26.813771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51355 (local address 127.0.97.62:42315)
0929 15:52:26.813951 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:26.813956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:26.813977 (+ 21us) server_negotiation.cc:407] Connection header received
0929 15:52:26.814049 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:26.814054 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:26.814136 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:26.814229 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:26.814967 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:26.815792 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:26.816874 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:26.817254 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:26.817393 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:26.820673 (+ 3280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:51355: BlockingRecv error: recv got EOF from 127.0.97.3:51355 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 29 15:52:26 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:26 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:26.828850 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:26.821310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.3:39779)
0929 15:52:26.821487 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:26.821502 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:26.821598 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:26.821940 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:26.821945 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:26.822228 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:26.822356 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:26.822363 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:26.823285 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:26.823291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:26.824176 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:26.824185 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:26.824834 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:26.824839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:26.824999 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:26.825507 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:26.825522 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:26.828533 (+ 3011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":63}
W20260929 15:52:26.828960 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:26.821466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39779 (local address 127.0.97.62:42315)
0929 15:52:26.821704 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:26.821709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:26.821723 (+ 14us) server_negotiation.cc:407] Connection header received
0929 15:52:26.821768 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:26.821771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:26.821822 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:26.821919 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:26.822483 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:26.823154 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:26.824315 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:26.824700 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:26.824823 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:26.828735 (+ 3912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:39779: BlockingRecv error: recv got EOF from 127.0.97.3:39779 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
W20260929 15:52:26.829246 9618 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 15:52:26 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:26 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:26.973064 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:26.964406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.1:41233)
0929 15:52:26.964963 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:26.964979 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:26.965098 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:26.965494 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:26.965498 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:26.965959 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:26.966146 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:26.966154 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:26.967169 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:26.967174 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:26.968140 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:26.968151 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:26.968897 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:26.968902 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:26.969047 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:26.969710 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:26.969736 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:26.972845 (+ 3109us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":323,"threads_started":1}
W20260929 15:52:26.973191 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:26.964513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41233 (local address 127.0.97.62:42315)
0929 15:52:26.964682 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:26.964687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:26.965053 (+ 366us) server_negotiation.cc:407] Connection header received
0929 15:52:26.965223 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:26.965228 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:26.965289 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:26.965472 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:26.966276 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:26.967025 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:26.968287 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:26.968714 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:26.968843 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:26.973029 (+ 4186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:41233: BlockingRecv error: recv got EOF from 127.0.97.1:41233 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 15:52:27 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:27 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:27.408133 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:27.399693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.2:55253)
0929 15:52:27.400223 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:27.400242 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:27.400405 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:27.400835 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:27.400840 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:27.401474 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:27.401733 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:27.401746 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:27.402667 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:27.402671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:27.403757 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:27.403768 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:27.404439 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:27.404444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:27.404557 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:27.405166 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:27.405183 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:27.407904 (+ 2721us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":284,"threads_started":1}
W20260929 15:52:27.408384 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:27.399757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55253 (local address 127.0.97.62:42315)
0929 15:52:27.400036 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:27.400044 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:27.400346 (+ 302us) server_negotiation.cc:407] Connection header received
0929 15:52:27.400531 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:27.400540 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:27.400633 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:27.400756 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:27.401881 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:27.402552 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:27.403875 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:27.404292 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:27.404380 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:27.408113 (+ 3733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:55253: BlockingRecv error: recv got EOF from 127.0.97.2:55253 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Sep 29 15:52:27 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:27 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:27.841586 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:27.829812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.3:33697)
0929 15:52:27.830314 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:27.830331 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:27.830441 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:27.830773 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:27.830776 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:27.831204 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:27.831409 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:27.831419 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:27.832366 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:27.832370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:27.833289 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:27.833299 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:27.834039 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:27.834043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:27.834160 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:27.838515 (+ 4355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:27.838538 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:27.841303 (+ 2765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":275,"threads_started":1}
W20260929 15:52:27.841725 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:27.829880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33697 (local address 127.0.97.62:42315)
0929 15:52:27.830065 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:27.830070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:27.830402 (+ 332us) server_negotiation.cc:407] Connection header received
0929 15:52:27.830531 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:27.830535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:27.830591 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:27.830725 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:27.831531 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:27.832246 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:27.833484 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:27.833890 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:27.833990 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:27.841525 (+ 7535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:33697: BlockingRecv error: recv got EOF from 127.0.97.3:33697 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 29 15:52:27 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:27 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:27.981949 9626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:27.974023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.1:59965)
0929 15:52:27.974541 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:27.974556 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:27.974665 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:27.974972 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:27.974975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:27.975347 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:27.975493 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:27.975501 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:27.976437 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:27.976440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:27.977315 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:27.977325 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:27.978022 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:27.978026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:27.978129 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:27.978680 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:27.978696 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:27.981595 (+ 2899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":302,"threads_started":1}
W20260929 15:52:27.982023 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:27.974116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59965 (local address 127.0.97.62:42315)
0929 15:52:27.974276 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:27.974282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:27.974625 (+ 343us) server_negotiation.cc:407] Connection header received
0929 15:52:27.974753 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:27.974756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:27.974850 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:27.974933 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:27.975604 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:27.976331 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:27.977490 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:27.977898 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:27.977990 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:27.981773 (+ 3783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:59965: BlockingRecv error: recv got EOF from 127.0.97.1:59965 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 15:52:28 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:28 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:28.417506 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:28.408992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.2:46095)
0929 15:52:28.409495 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:28.409512 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:28.409626 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:28.409965 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:28.409969 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:28.410353 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:28.410497 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:28.410506 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:28.411590 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:28.411596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:28.412913 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:28.412926 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:28.413729 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:28.413735 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:28.413889 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:28.414463 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:28.414480 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:28.417139 (+ 2659us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":243,"threads_started":1}
W20260929 15:52:28.417573 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:28.409032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46095 (local address 127.0.97.62:42315)
0929 15:52:28.409224 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:28.409235 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:28.409593 (+ 358us) server_negotiation.cc:407] Connection header received
0929 15:52:28.409721 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:28.409725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:28.409796 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:28.409886 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:28.410616 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:28.411454 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:28.413064 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:28.413582 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:28.413714 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:28.417333 (+ 3619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:46095: BlockingRecv error: recv got EOF from 127.0.97.2:46095 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 29 15:52:28 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:28 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:28.856729 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:28.842494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.3:46633)
0929 15:52:28.843223 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:28.843239 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:28.843354 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:28.843624 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:28.843627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:28.843975 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:28.844112 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:28.844120 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:28.845097 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:28.845101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:28.846114 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:28.846126 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:28.846898 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:28.846904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:28.847075 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:28.853196 (+ 6121us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:28.853216 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:28.856336 (+ 3120us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":263,"threads_started":1}
W20260929 15:52:28.856820 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:28.842709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46633 (local address 127.0.97.62:42315)
0929 15:52:28.842917 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:28.842926 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:28.843316 (+ 390us) server_negotiation.cc:407] Connection header received
0929 15:52:28.843446 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:28.843450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:28.843508 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:28.843585 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:28.844237 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:28.844956 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:28.846275 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:28.846744 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:28.846886 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:28.856582 (+ 9696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:46633: BlockingRecv error: recv got EOF from 127.0.97.3:46633 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 15:52:28 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:28 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:28.990630 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:28.982871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.1:52123)
0929 15:52:28.983310 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:28.983326 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:28.983446 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:28.983686 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:28.983689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:28.984000 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:28.984156 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:28.984164 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:28.985125 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:28.985129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:28.986050 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:28.986060 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:28.986678 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:28.986682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:28.986796 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:28.987331 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:28.987355 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:28.990275 (+ 2920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":233,"threads_started":1}
W20260929 15:52:28.990666 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:28.983019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52123 (local address 127.0.97.62:42315)
0929 15:52:28.983230 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:28.983236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:28.983419 (+ 183us) server_negotiation.cc:407] Connection header received
0929 15:52:28.983538 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:28.983542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:28.983598 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:28.983673 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:28.984271 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:28.985012 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:28.986160 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:28.986560 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:28.986647 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:28.990445 (+ 3798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:52123: BlockingRecv error: recv got EOF from 127.0.97.1:52123 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 15:52:29 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:29 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:29.426687 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:29.418505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.2:34553)
0929 15:52:29.419000 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:29.419015 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:29.419214 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:29.419569 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:29.419573 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:29.419959 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:29.420121 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:29.420130 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:29.421146 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:29.421154 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:29.422064 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:29.422076 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:29.422746 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:29.422751 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:29.422928 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:29.423526 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:29.423544 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:29.426337 (+ 2793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":280,"threads_started":1}
W20260929 15:52:29.426759 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:29.418560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34553 (local address 127.0.97.62:42315)
0929 15:52:29.418848 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:29.418855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:29.419127 (+ 272us) server_negotiation.cc:407] Connection header received
0929 15:52:29.419355 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:29.419360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:29.419429 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:29.419554 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:29.420280 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:29.421017 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:29.422208 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:29.422605 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:29.422735 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:29.426523 (+ 3788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:34553: BlockingRecv error: recv got EOF from 127.0.97.2:34553 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Sep 29 15:52:29 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:29 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:29.865639 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:29.857834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.3:53691)
0929 15:52:29.858289 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:29.858306 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:29.858417 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:29.858774 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:29.858777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:29.859110 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:29.859272 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:29.859280 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:29.860292 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:29.860296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:29.861185 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:29.861196 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:29.861878 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:29.861895 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:29.862011 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:29.862666 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:29.862684 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:29.865332 (+ 2648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":253,"threads_started":1}
W20260929 15:52:29.865705 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:29.857900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53691 (local address 127.0.97.62:42315)
0929 15:52:29.858138 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:29.858144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:29.858391 (+ 247us) server_negotiation.cc:407] Connection header received
0929 15:52:29.858558 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:29.858562 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:29.858630 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:29.858723 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:29.859409 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:29.860156 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:29.861315 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:29.861753 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:29.861852 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:29.865537 (+ 3685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:53691: BlockingRecv error: recv got EOF from 127.0.97.3:53691 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 15:52:29 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:29 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:29.999941 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:29.991621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.1:44567)
0929 15:52:29.992149 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:29.992165 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:29.992337 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:29.992670 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:29.992675 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:29.993055 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:29.993197 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:29.993205 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:29.994240 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:29.994247 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:29.995198 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:29.995210 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:29.995949 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:29.995955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:29.996140 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:29.996717 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:29.996735 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:29.999708 (+ 2973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":305,"threads_started":1}
W20260929 15:52:30.000298 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:29.991702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44567 (local address 127.0.97.62:42315)
0929 15:52:29.992007 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:29.992015 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:29.992259 (+ 244us) server_negotiation.cc:407] Connection header received
0929 15:52:29.992462 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:29.992467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:29.992533 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:29.992640 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:29.993375 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:29.994104 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:29.995349 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:29.995788 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:29.995917 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:29.999912 (+ 3995us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:44567: BlockingRecv error: recv got EOF from 127.0.97.1:44567 (error 108)
Metrics: {"server-negotiator.queue_time_us":145}
Sep 29 15:52:30 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:30 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:30.436587 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:30.427659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.2:48175)
0929 15:52:30.428165 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:30.428181 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:30.428297 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:30.428717 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:30.428720 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:30.429238 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:30.429516 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:30.429529 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:30.430724 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:30.430728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:30.431772 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:30.431785 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:30.432621 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:30.432626 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:30.432782 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:30.433514 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:30.433532 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:30.436361 (+ 2829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":263,"threads_started":1}
W20260929 15:52:30.437005 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:30.427691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48175 (local address 127.0.97.62:42315)
0929 15:52:30.427875 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:30.427885 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:30.428321 (+ 436us) server_negotiation.cc:407] Connection header received
0929 15:52:30.428421 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:30.428426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:30.428515 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:30.428662 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:30.429671 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:30.430526 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:30.431993 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:30.432456 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:30.432577 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:30.436795 (+ 4218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:48175: BlockingRecv error: recv got EOF from 127.0.97.2:48175 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 29 15:52:30 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:30 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 15:52:30.877775 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:30.866783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60419 (local address 127.0.97.62:42315)
0929 15:52:30.867063 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:30.867069 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:30.867280 (+ 211us) server_negotiation.cc:407] Connection header received
0929 15:52:30.867508 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:30.867521 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:30.867601 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:30.867715 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:30.868478 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:30.869280 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:30.870740 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:30.871489 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:30.871636 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:30.877485 (+ 5849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:60419: BlockingRecv error: recv got EOF from 127.0.97.3:60419 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
I20260929 15:52:30.877854 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:30.866580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.3:60419)
0929 15:52:30.867175 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:30.867192 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:30.867368 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:30.867744 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:30.867749 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:30.868167 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:30.868318 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:30.868326 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:30.869486 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:30.869492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:30.870533 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:30.870544 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:30.871650 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:30.871656 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:30.871875 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:30.872732 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:30.872756 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:30.877243 (+ 4487us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":354,"threads_started":1}
Sep 29 15:52:31 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:31 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:31.011093 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:31.000920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.1:56985)
0929 15:52:31.001555 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:31.001575 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:31.001813 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:31.002372 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:31.002376 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:31.002973 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:31.003294 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:31.003308 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:31.004363 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:31.004367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:31.005334 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:31.005380 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:31.006206 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:31.006212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:31.006390 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:31.007153 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:31.007173 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:31.010725 (+ 3552us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":258,"threads_started":1}
W20260929 15:52:31.011233 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:31.000972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56985 (local address 127.0.97.62:42315)
0929 15:52:31.001160 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:31.001165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:31.001679 (+ 514us) server_negotiation.cc:407] Connection header received
0929 15:52:31.002077 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:31.002082 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:31.002145 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:31.002248 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:31.003494 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:31.004205 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:31.005499 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:31.005974 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:31.006081 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:31.010975 (+ 4894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:56985: BlockingRecv error: recv got EOF from 127.0.97.1:56985 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 15:52:31 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:31 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:31.446575 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:31.437934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.2:58905)
0929 15:52:31.438629 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:31.438648 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:31.438854 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:31.439218 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:31.439226 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:31.439680 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:31.439850 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:31.439859 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:31.440890 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:31.440894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:31.441870 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:31.441882 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:31.442572 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:31.442578 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:31.442745 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:31.443372 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:31.443389 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:31.446234 (+ 2845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":325,"threads_started":1}
W20260929 15:52:31.446851 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:31.438220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58905 (local address 127.0.97.62:42315)
0929 15:52:31.438488 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:31.438497 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:31.438736 (+ 239us) server_negotiation.cc:407] Connection header received
0929 15:52:31.439005 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:31.439011 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:31.439077 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:31.439179 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:31.439995 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:31.440718 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:31.442015 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:31.442425 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:31.442555 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:31.446583 (+ 4028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58905: BlockingRecv error: recv got EOF from 127.0.97.2:58905 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Sep 29 15:52:31 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:31 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:31.889063 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:31.879096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.3:53661)
0929 15:52:31.879987 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:31.880015 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:31.880199 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:31.880702 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:31.880708 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:31.881282 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:31.881555 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:31.881566 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:31.882693 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:31.882697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:31.883630 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:31.883641 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:31.884380 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:31.884386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:31.884598 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:31.885517 (+ 919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:31.885538 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:31.888816 (+ 3278us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":517,"threads_started":1}
W20260929 15:52:31.889292 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:31.879379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53661 (local address 127.0.97.62:42315)
0929 15:52:31.879652 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:31.879658 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:31.880169 (+ 511us) server_negotiation.cc:407] Connection header received
0929 15:52:31.880326 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:31.880332 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:31.880426 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:31.880590 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:31.881775 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:31.882553 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:31.883765 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:31.884172 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:31.884269 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:31.889019 (+ 4750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:53661: BlockingRecv error: recv got EOF from 127.0.97.3:53661 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 29 15:52:32 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:32 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:32.022171 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:32.012360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.1:55299)
0929 15:52:32.013087 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:32.013102 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:32.013273 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:32.013674 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:32.013679 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:32.014074 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:32.014218 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:32.014226 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:32.015252 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:32.015259 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:32.016321 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:32.016337 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:32.017473 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:32.017551 (+ 78us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:32.017769 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:32.018609 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:32.018625 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:32.021952 (+ 3327us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":484,"threads_started":1}
W20260929 15:52:32.022687 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:32.012596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55299 (local address 127.0.97.62:42315)
0929 15:52:32.012841 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:32.012851 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:32.013196 (+ 345us) server_negotiation.cc:407] Connection header received
0929 15:52:32.013430 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:32.013438 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:32.013532 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:32.013643 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:32.014360 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:32.015113 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:32.016554 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:32.017196 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:32.017301 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:32.022381 (+ 5080us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55299: BlockingRecv error: recv got EOF from 127.0.97.1:55299 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Sep 29 15:52:32 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:32 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:32.456159 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:32.447663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.2:40387)
0929 15:52:32.448395 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:32.448415 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:32.448551 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:32.448920 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:32.448924 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:32.449328 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:32.449491 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:32.449500 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:32.450449 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:32.450457 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:32.451373 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:32.451384 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:32.452053 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:32.452059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:32.452233 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:32.452890 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:32.452909 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:32.455846 (+ 2937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":465,"threads_started":1}
W20260929 15:52:32.456336 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:32.447711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40387 (local address 127.0.97.62:42315)
0929 15:52:32.448008 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:32.448017 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:32.448486 (+ 469us) server_negotiation.cc:407] Connection header received
0929 15:52:32.448701 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:32.448708 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:32.448785 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:32.448888 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:32.449629 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:32.450318 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:32.451516 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:32.451904 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:32.452038 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:32.456069 (+ 4031us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:40387: BlockingRecv error: recv got EOF from 127.0.97.2:40387 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 29 15:52:32 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:32 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:32.902742 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:32.890111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.3:45633)
0929 15:52:32.890797 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:32.890814 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:32.891002 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:32.891341 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:32.891345 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:32.891860 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:32.892033 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:32.892041 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:32.893104 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:32.893109 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:32.894665 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:32.894676 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:32.895417 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:32.895422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:32.895606 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:32.899083 (+ 3477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:32.899109 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:32.902451 (+ 3342us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":371,"threads_started":1}
W20260929 15:52:32.903056 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:32.890279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45633 (local address 127.0.97.62:42315)
0929 15:52:32.890560 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:32.890570 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:32.890927 (+ 357us) server_negotiation.cc:407] Connection header received
0929 15:52:32.891123 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:32.891129 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:32.891199 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:32.891313 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:32.892171 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:32.892969 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:32.894816 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:32.895253 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:32.895405 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:32.902738 (+ 7333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:45633: BlockingRecv error: recv got EOF from 127.0.97.3:45633 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 29 15:52:33 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:33 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:33.032881 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:33.023264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.1:57557)
0929 15:52:33.024056 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:33.024082 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:33.024230 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:33.024694 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:33.024699 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:33.025134 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:33.025317 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:33.025325 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:33.026470 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:33.026474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:33.027492 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:33.027503 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:33.028343 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:33.028348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:33.028471 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:33.029055 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:33.029072 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:33.032627 (+ 3555us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":353,"threads_started":1}
W20260929 15:52:33.033089 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:33.023425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57557 (local address 127.0.97.62:42315)
0929 15:52:33.023748 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:33.023756 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:33.024218 (+ 462us) server_negotiation.cc:407] Connection header received
0929 15:52:33.024390 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:33.024395 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:33.024471 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:33.024580 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:33.025544 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:33.026309 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:33.027672 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:33.028189 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:33.028285 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:33.032864 (+ 4579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:57557: BlockingRecv error: recv got EOF from 127.0.97.1:57557 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Sep 29 15:52:33 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:33 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:33.467972 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:33.457510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.2:55703)
0929 15:52:33.458241 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:33.458260 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:33.458460 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:33.458850 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:33.458855 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:33.459356 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:33.459569 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:33.459580 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:33.460934 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:33.460942 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:33.462090 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:33.462111 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:33.463238 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:33.463243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:33.463367 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:33.464114 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:33.464132 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:33.467612 (+ 3480us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":327,"threads_started":1}
W20260929 15:52:33.468078 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:33.457579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55703 (local address 127.0.97.62:42315)
0929 15:52:33.457805 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:33.457811 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:33.458365 (+ 554us) server_negotiation.cc:407] Connection header received
0929 15:52:33.458610 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:33.458616 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:33.458693 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:33.458807 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:33.459782 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:33.460688 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:33.462345 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:33.463026 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:33.463167 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:33.467825 (+ 4658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:55703: BlockingRecv error: recv got EOF from 127.0.97.2:55703 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 29 15:52:33 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:33 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:33.913959 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:33.903825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.3:59741)
0929 15:52:33.904701 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:33.904721 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:33.904885 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:33.905465 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:33.905474 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:33.906042 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:33.906238 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:33.906246 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:33.907516 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:33.907520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:33.909143 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:33.909155 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:33.910008 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:33.910012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:33.910116 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:33.910978 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:33.910995 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:33.913706 (+ 2711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":349,"threads_started":1}
W20260929 15:52:33.914407 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:33.904138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59741 (local address 127.0.97.62:42315)
0929 15:52:33.904352 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:33.904358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:33.904872 (+ 514us) server_negotiation.cc:407] Connection header received
0929 15:52:33.905092 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:33.905097 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:33.905181 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:33.905278 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:33.906418 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:33.907345 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:33.909308 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:33.909856 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:33.909958 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:33.913983 (+ 4025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:59741: BlockingRecv error: recv got EOF from 127.0.97.3:59741 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 15:52:34 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:34 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:34.042748 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:34.034031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.1:58315)
0929 15:52:34.034580 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:34.034596 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:34.034705 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:34.035008 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:34.035011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:34.035379 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:34.035530 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:34.035537 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:34.036616 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:34.036620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:34.037554 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:34.037565 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:34.038270 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:34.038274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:34.038391 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:34.039180 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:34.039197 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:34.042416 (+ 3219us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":302,"threads_started":1}
W20260929 15:52:34.042851 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:34.034213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58315 (local address 127.0.97.62:42315)
0929 15:52:34.034406 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:34.034411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:34.034674 (+ 263us) server_negotiation.cc:407] Connection header received
0929 15:52:34.034799 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:34.034803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:34.034866 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:34.034982 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:34.035677 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:34.036476 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:34.037676 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:34.038106 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:34.038205 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:34.042601 (+ 4396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:58315: BlockingRecv error: recv got EOF from 127.0.97.1:58315 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 29 15:52:34 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:34 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:34.479087 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:34.469077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.2:33755)
0929 15:52:34.469610 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:34.469627 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:34.469786 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:34.470273 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:34.470278 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:34.470865 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:34.471085 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:34.471098 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:34.472659 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:34.472668 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:34.473690 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:34.473702 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:34.474417 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:34.474423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:34.474601 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:34.475214 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:34.475233 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:34.478684 (+ 3451us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":302,"threads_started":1}
W20260929 15:52:34.479166 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:34.469321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33755 (local address 127.0.97.62:42315)
0929 15:52:34.469540 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:34.469548 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:34.469721 (+ 173us) server_negotiation.cc:407] Connection header received
0929 15:52:34.470006 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:34.470012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:34.470088 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:34.470211 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:34.471248 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:34.472518 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:34.473827 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:34.474238 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:34.474379 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:34.478937 (+ 4558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:33755: BlockingRecv error: recv got EOF from 127.0.97.2:33755 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 29 15:52:34 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:34 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:34.924009 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:34.914870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.3:38835)
0929 15:52:34.915324 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:34.915340 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:34.915461 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:34.915764 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:34.915768 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:34.916192 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:34.916335 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:34.916343 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:34.917401 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:34.917405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:34.918496 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:34.918509 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:34.919262 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:34.919268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:34.919456 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:34.920285 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:34.920308 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:34.923642 (+ 3334us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":263,"threads_started":1}
W20260929 15:52:34.924129 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:34.914969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38835 (local address 127.0.97.62:42315)
0929 15:52:34.915230 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:34.915237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:34.915422 (+ 185us) server_negotiation.cc:407] Connection header received
0929 15:52:34.915556 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:34.915559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:34.915617 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:34.915698 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:34.916518 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:34.917218 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:34.918663 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:34.919081 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:34.919226 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:34.923850 (+ 4624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:38835: BlockingRecv error: recv got EOF from 127.0.97.3:38835 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 29 15:52:35 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:35 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:35.054908 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:35.043993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.1:40457)
0929 15:52:35.044735 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:35.044763 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:35.044952 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:35.045679 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:35.045686 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:35.046222 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:35.046433 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:35.046442 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:35.047756 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:35.047761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:35.049622 (+ 1861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:35.049636 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:35.050431 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:35.050437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:35.050605 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:35.051266 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:35.051290 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:35.054536 (+ 3246us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":331,"threads_started":1}
W20260929 15:52:35.054975 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:35.044121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40457 (local address 127.0.97.62:42315)
0929 15:52:35.044419 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:35.044428 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:35.044867 (+ 439us) server_negotiation.cc:407] Connection header received
0929 15:52:35.045157 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:35.045163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:35.045283 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:35.045451 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:35.046700 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:35.047551 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:35.049777 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:35.050265 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:35.050400 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:35.054730 (+ 4330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:40457: BlockingRecv error: recv got EOF from 127.0.97.1:40457 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 29 15:52:35 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:35 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:35.489565 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:35.480298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.2:59435)
0929 15:52:35.481124 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:35.481141 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:35.481267 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:35.481620 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:35.481623 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:35.481995 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:35.482162 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:35.482170 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:35.483217 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:35.483220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:35.484177 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:35.484189 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:35.484884 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:35.484890 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:35.485019 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:35.485673 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:35.485697 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:35.489286 (+ 3589us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":553,"threads_started":1}
W20260929 15:52:35.489813 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:35.480350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59435 (local address 127.0.97.62:42315)
0929 15:52:35.480563 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:35.480571 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:35.481230 (+ 659us) server_negotiation.cc:407] Connection header received
0929 15:52:35.481422 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:35.481426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:35.481487 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:35.481568 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:35.482329 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:35.483076 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:35.484317 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:35.484727 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:35.484826 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:35.489558 (+ 4732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:59435: BlockingRecv error: recv got EOF from 127.0.97.2:59435 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 29 15:52:35 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:35 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:35.935441 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:35.925543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.3:45879)
0929 15:52:35.926382 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:35.926400 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:35.926603 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:35.927017 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:35.927024 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:35.927489 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:35.927682 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:35.927692 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:35.928800 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:35.928808 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:35.930281 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:35.930315 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:35.931293 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:35.931299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:35.931480 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:35.932286 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:35.932304 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:35.935189 (+ 2885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":712,"thread_start_us":552,"threads_started":1}
W20260929 15:52:35.935626 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:35.925566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45879 (local address 127.0.97.62:42315)
0929 15:52:35.925882 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:35.925894 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:35.926515 (+ 621us) server_negotiation.cc:407] Connection header received
0929 15:52:35.926758 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:35.926762 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:35.926830 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:35.926941 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:35.927839 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:35.928648 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:35.930479 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:35.931130 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:35.931279 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:35.935390 (+ 4111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:45879: BlockingRecv error: recv got EOF from 127.0.97.3:45879 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 29 15:52:36 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:36 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:36.066767 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:36.056211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.1:49353)
0929 15:52:36.056704 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:36.056724 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:36.056871 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:36.057217 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:36.057221 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:36.057649 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:36.057875 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:36.057891 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:36.059380 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:36.059386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:36.060539 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:36.060558 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:36.061504 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:36.061514 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:36.061744 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:36.062592 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:36.062614 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:36.066366 (+ 3752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":283,"threads_started":1}
W20260929 15:52:36.066987 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:36.056265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49353 (local address 127.0.97.62:42315)
0929 15:52:36.056471 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:36.056479 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:36.056816 (+ 337us) server_negotiation.cc:407] Connection header received
0929 15:52:36.057005 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:36.057010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:36.057083 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:36.057189 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:36.058128 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:36.059182 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:36.060735 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:36.061158 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:36.061259 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:36.066630 (+ 5371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:49353: BlockingRecv error: recv got EOF from 127.0.97.1:49353 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 15:52:36 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:36 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:36.498329 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:36.490575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.2:49127)
0929 15:52:36.491158 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:36.491179 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:36.491315 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:36.491618 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:36.491622 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:36.491969 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:36.492105 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:36.492112 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:36.493022 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:36.493026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:36.493955 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:36.493965 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:36.494608 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:36.494612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:36.494723 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:36.495364 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:36.495382 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:36.498024 (+ 2642us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":340,"threads_started":1}
W20260929 15:52:36.498441 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:36.490662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49127 (local address 127.0.97.62:42315)
0929 15:52:36.490886 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:36.490895 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:36.491269 (+ 374us) server_negotiation.cc:407] Connection header received
0929 15:52:36.491443 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:36.491446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:36.491503 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:36.491587 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:36.492225 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:36.492912 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:36.494069 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:36.494491 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:36.494581 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:36.498217 (+ 3636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:49127: BlockingRecv error: recv got EOF from 127.0.97.2:49127 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 29 15:52:36 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:36 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:36.945062 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:36.936414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.3:42525)
0929 15:52:36.936907 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:36.936923 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:36.937027 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:36.937303 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:36.937306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:36.937683 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:36.937838 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:36.937846 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:36.939030 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:36.939035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:36.940120 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:36.940130 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:36.940768 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:36.940772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:36.940886 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:36.941486 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:36.941504 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:36.944794 (+ 3290us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":313,"threads_started":1}
W20260929 15:52:36.945212 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:36.936533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42525 (local address 127.0.97.62:42315)
0929 15:52:36.936830 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:36.936836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:36.936998 (+ 162us) server_negotiation.cc:407] Connection header received
0929 15:52:36.937120 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:36.937124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:36.937181 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:36.937270 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:36.938001 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:36.938915 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:36.940252 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:36.940625 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:36.940742 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:36.945032 (+ 4290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:42525: BlockingRecv error: recv got EOF from 127.0.97.3:42525 (error 108)
Metrics: {"server-negotiator.queue_time_us":148}
Sep 29 15:52:37 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:37 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:37.076376 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:37.067903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.1:35183)
0929 15:52:37.068429 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:37.068450 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:37.068573 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:37.068974 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:37.068979 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:37.069451 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:37.069632 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:37.069640 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:37.070561 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:37.070564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:37.071589 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:37.071600 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:37.072334 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:37.072342 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:37.072526 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:37.073160 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:37.073184 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:37.076129 (+ 2945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":259,"threads_started":1}
W20260929 15:52:37.076522 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:37.067994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35183 (local address 127.0.97.62:42315)
0929 15:52:37.068196 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:37.068205 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:37.068510 (+ 305us) server_negotiation.cc:407] Connection header received
0929 15:52:37.068721 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:37.068725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:37.068826 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:37.068948 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:37.069745 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:37.070449 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:37.071700 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:37.072129 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:37.072221 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:37.076328 (+ 4107us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:35183: BlockingRecv error: recv got EOF from 127.0.97.1:35183 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 15:52:37 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:37 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:37.509455 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:37.499239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.2:50685)
0929 15:52:37.499785 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:37.499801 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:37.499967 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:37.500315 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:37.500319 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:37.500725 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:37.500896 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:37.500904 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:37.502259 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:37.502266 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:37.503453 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:37.503678 (+ 225us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:37.504580 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:37.504586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:37.504807 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:37.505643 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:37.505663 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:37.509026 (+ 3363us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":290,"threads_started":1}
W20260929 15:52:37.509485 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:37.499392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50685 (local address 127.0.97.62:42315)
0929 15:52:37.499646 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:37.499653 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:37.499888 (+ 235us) server_negotiation.cc:407] Connection header received
0929 15:52:37.500095 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:37.500100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:37.500167 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:37.500268 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:37.501034 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:37.502092 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:37.503896 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:37.504397 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:37.504553 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:37.509231 (+ 4678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:50685: BlockingRecv error: recv got EOF from 127.0.97.2:50685 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 29 15:52:37 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:37 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:37.958963 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:37.946269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.3:33603)
0929 15:52:37.946759 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:37.946774 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:37.946933 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:37.947359 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:37.947362 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:37.947742 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:37.947983 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:37.947991 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:37.949015 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:37.949019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:37.949954 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:37.949965 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:37.950879 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:37.950884 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:37.951009 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:37.955848 (+ 4839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:37.955867 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:37.958703 (+ 2836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":254,"threads_started":1}
W20260929 15:52:37.959151 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:37.946442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33603 (local address 127.0.97.62:42315)
0929 15:52:37.946669 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:37.946678 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:37.946867 (+ 189us) server_negotiation.cc:407] Connection header received
0929 15:52:37.947162 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:37.947167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:37.947230 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:37.947329 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:37.948129 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:37.948870 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:37.950186 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:37.950674 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:37.950821 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:37.958949 (+ 8128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:33603: BlockingRecv error: recv got EOF from 127.0.97.3:33603 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 29 15:52:38 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:38 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:38.086393 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:38.077633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.1:53113)
0929 15:52:38.078281 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:38.078297 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:38.078445 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:38.078759 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:38.078763 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:38.079173 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:38.079314 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:38.079321 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:38.080367 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:38.080373 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:38.081276 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:38.081286 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:38.082040 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:38.082046 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:38.082227 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:38.082855 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:38.082872 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:38.086137 (+ 3265us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":311,"threads_started":1}
W20260929 15:52:38.086932 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:38.077612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53113 (local address 127.0.97.62:42315)
0929 15:52:38.077797 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:38.077802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:38.078391 (+ 589us) server_negotiation.cc:407] Connection header received
0929 15:52:38.078561 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:38.078566 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:38.078632 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:38.078728 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:38.079457 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:38.080235 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:38.081467 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:38.081898 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:38.082023 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:38.086586 (+ 4563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:53113: BlockingRecv error: recv got EOF from 127.0.97.1:53113 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 15:52:38 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:38 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:38.518649 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:38.510444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.2:45135)
0929 15:52:38.510992 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:38.511008 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:38.511120 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:38.511446 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:38.511450 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:38.511831 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:38.511999 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:38.512007 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:38.512959 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:38.512963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:38.514046 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:38.514057 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:38.514730 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:38.514735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:38.514853 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:38.515462 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:38.515480 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:38.518386 (+ 2906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":308,"threads_started":1}
W20260929 15:52:38.519160 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:38.510517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45135 (local address 127.0.97.62:42315)
0929 15:52:38.510686 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:38.510692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:38.511104 (+ 412us) server_negotiation.cc:407] Connection header received
0929 15:52:38.511220 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:38.511224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:38.511284 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:38.511381 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:38.512127 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:38.512832 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:38.514172 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:38.514609 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:38.514701 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:38.518807 (+ 4106us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:45135: BlockingRecv error: recv got EOF from 127.0.97.2:45135 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 15:52:38 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:38 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:38.967839 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:38.959964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.3:56977)
0929 15:52:38.960506 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:38.960527 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:38.960660 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:38.961006 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:38.961010 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:38.961475 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:38.961627 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:38.961634 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:38.962576 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:38.962581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:38.963499 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:38.963510 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:38.964178 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:38.964182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:38.964304 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:38.964928 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:38.964945 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:38.967618 (+ 2673us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":309,"threads_started":1}
W20260929 15:52:38.968346 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:38.959998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56977 (local address 127.0.97.62:42315)
0929 15:52:38.960192 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:38.960201 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:38.960612 (+ 411us) server_negotiation.cc:407] Connection header received
0929 15:52:38.960767 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:38.960774 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:38.960850 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:38.960940 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:38.961745 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:38.962426 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:38.963627 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:38.964053 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:38.964143 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:38.967838 (+ 3695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:56977: BlockingRecv error: recv got EOF from 127.0.97.3:56977 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 15:52:39 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:39 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:39.095630 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:39.087618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.1:51451)
0929 15:52:39.088155 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:39.088170 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:39.088318 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:39.088630 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:39.088634 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:39.089046 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:39.089211 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:39.089219 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:39.090340 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:39.090348 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:39.091239 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:39.091249 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:39.091992 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:39.091997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:39.092151 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:39.092741 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:39.092758 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:39.095407 (+ 2649us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":288,"threads_started":1}
W20260929 15:52:39.095788 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:39.087775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51451 (local address 127.0.97.62:42315)
0929 15:52:39.088034 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:39.088042 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:39.088252 (+ 210us) server_negotiation.cc:407] Connection header received
0929 15:52:39.088451 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:39.088455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:39.088511 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:39.088610 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:39.089381 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:39.090218 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:39.091376 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:39.091818 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:39.091960 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:39.095581 (+ 3621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:51451: BlockingRecv error: recv got EOF from 127.0.97.1:51451 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 29 15:52:39 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:39 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:39.527268 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:39.519508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.2:59303)
0929 15:52:39.520080 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:39.520096 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:39.520216 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:39.520500 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:39.520503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:39.520907 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:39.521079 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:39.521087 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:39.522108 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:39.522112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:39.523028 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:39.523038 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:39.523697 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:39.523701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:39.523808 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:39.524361 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:39.524377 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:39.526944 (+ 2567us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":296,"threads_started":1}
W20260929 15:52:39.527329 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:39.519624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59303 (local address 127.0.97.62:42315)
0929 15:52:39.519818 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:39.519823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:39.520189 (+ 366us) server_negotiation.cc:407] Connection header received
0929 15:52:39.520314 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:39.520318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:39.520377 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:39.520469 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:39.521188 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:39.521954 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:39.523155 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:39.523551 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:39.523639 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:39.527119 (+ 3480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:59303: BlockingRecv error: recv got EOF from 127.0.97.2:59303 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 29 15:52:39 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:39 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:39.977401 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:39.968766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.3:43289)
0929 15:52:39.969280 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:39.969295 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:39.969465 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:39.969806 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:39.969810 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:39.970216 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:39.970388 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:39.970397 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:39.971505 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:39.971509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:39.972656 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:39.972669 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:39.973446 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:39.973462 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:39.973624 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:39.974293 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:39.974317 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:39.977123 (+ 2806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":259,"threads_started":1}
W20260929 15:52:39.977551 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:39.968836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43289 (local address 127.0.97.62:42315)
0929 15:52:39.969090 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:39.969097 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:39.969389 (+ 292us) server_negotiation.cc:407] Connection header received
0929 15:52:39.969620 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:39.969623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:39.969680 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:39.969782 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:39.970520 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:39.971318 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:39.972799 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:39.973199 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:39.973309 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:39.977307 (+ 3998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:43289: BlockingRecv error: recv got EOF from 127.0.97.3:43289 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Sep 29 15:52:40 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:40 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:40.104238 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:40.096542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.1:53369)
0929 15:52:40.097129 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:40.097146 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:40.097256 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:40.097631 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:40.097634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:40.097996 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:40.098135 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:40.098142 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:40.099130 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:40.099133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:40.100038 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:40.100048 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:40.100683 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:40.100687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:40.100803 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:40.101391 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:40.101408 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:40.103998 (+ 2590us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":384,"threads_started":1}
W20260929 15:52:40.104635 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:40.096786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53369 (local address 127.0.97.62:42315)
0929 15:52:40.097089 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:40.097097 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:40.097221 (+ 124us) server_negotiation.cc:407] Connection header received
0929 15:52:40.097405 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:40.097409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:40.097470 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:40.097557 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:40.098266 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:40.099011 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:40.100167 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:40.100561 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:40.100655 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:40.104259 (+ 3604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:53369: BlockingRecv error: recv got EOF from 127.0.97.1:53369 (error 108)
Metrics: {"server-negotiator.queue_time_us":189}
Sep 29 15:52:40 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:40 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:40.536306 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:40.528442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.2:49567)
0929 15:52:40.528917 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:40.528934 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:40.529044 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:40.529323 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:40.529326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:40.529736 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:40.529873 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:40.529881 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:40.530955 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:40.530959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:40.531866 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:40.531876 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:40.532577 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:40.532582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:40.532715 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:40.533295 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:40.533312 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:40.536029 (+ 2717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":254,"threads_started":1}
W20260929 15:52:40.536468 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:40.528484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49567 (local address 127.0.97.62:42315)
0929 15:52:40.528642 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:40.528646 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:40.529002 (+ 356us) server_negotiation.cc:407] Connection header received
0929 15:52:40.529146 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:40.529149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:40.529207 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:40.529302 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:40.530068 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:40.530834 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:40.531986 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:40.532419 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:40.532527 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:40.536235 (+ 3708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:49567: BlockingRecv error: recv got EOF from 127.0.97.2:49567 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 15:52:40 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:40 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:40.987504 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:40.978599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.3:59453)
0929 15:52:40.979198 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:40.979222 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:40.979401 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:40.979925 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:40.979932 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:40.980459 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:40.980668 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:40.980682 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:40.982052 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:40.982057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:40.982978 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:40.982989 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:40.983682 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:40.983687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:40.983854 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:40.984428 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:40.984445 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:40.987128 (+ 2683us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":302,"threads_started":1}
W20260929 15:52:40.987586 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:40.978638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59453 (local address 127.0.97.62:42315)
0929 15:52:40.978915 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:40.978926 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:40.979334 (+ 408us) server_negotiation.cc:407] Connection header received
0929 15:52:40.979554 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:40.979559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:40.979680 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:40.979811 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:40.980812 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:40.981871 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:40.983117 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:40.983539 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:40.983665 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:40.987349 (+ 3684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:59453: BlockingRecv error: recv got EOF from 127.0.97.3:59453 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 29 15:52:41 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:41 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:41.114078 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:41.105440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.1:34627)
0929 15:52:41.105938 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:41.105963 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:41.106065 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:41.106348 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:41.106351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:41.106732 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:41.106893 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:41.106901 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:41.108007 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:41.108015 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:41.109488 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:41.109501 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:41.110432 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:41.110438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:41.110610 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:41.111177 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:41.111194 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:41.113838 (+ 2644us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":288,"threads_started":1}
W20260929 15:52:41.114208 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:41.105527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34627 (local address 127.0.97.62:42315)
0929 15:52:41.105711 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:41.105716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:41.106032 (+ 316us) server_negotiation.cc:407] Connection header received
0929 15:52:41.106155 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:41.106158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:41.106227 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:41.106342 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:41.107036 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:41.107852 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:41.109641 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:41.110269 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:41.110418 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:41.114011 (+ 3593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:34627: BlockingRecv error: recv got EOF from 127.0.97.1:34627 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 29 15:52:41 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:52:41 dist-test-slave-r0ck krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:52:41.545703 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:41.537479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42315 (local address 127.0.97.2:47635)
0929 15:52:41.537924 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:41.537939 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:41.538047 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:41.538357 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:41.538361 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:41.538761 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:41.538932 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:41.538940 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:41.539885 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:41.539888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:41.540779 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:41.540789 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:41.541793 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:41.541798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:41.541962 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:41.542598 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:41.542615 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:41.545413 (+ 2798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42315: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":254,"threads_started":1}
W20260929 15:52:41.545773 9357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:41.537616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47635 (local address 127.0.97.62:42315)
0929 15:52:41.537837 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:41.537844 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:41.538010 (+ 166us) server_negotiation.cc:407] Connection header received
0929 15:52:41.538145 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:41.538148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:41.538204 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:41.538286 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:41.539060 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:41.539771 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:41.540940 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:41.541654 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:41.541783 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:41.545586 (+ 3803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:47635: BlockingRecv error: recv got EOF from 127.0.97.2:47635 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 15:52:41.809196 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 9208
I20260929 15:52:41.821623 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:41.810297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44792 (local address 127.0.97.1:38521)
0929 15:52:41.810941 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:41.810951 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:41.811260 (+ 309us) server_negotiation.cc:407] Connection header received
0929 15:52:41.811523 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:41.811532 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:41.811888 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:41.812091 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:41.813726 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:41.814725 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:41.815840 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:41.816346 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:41.816440 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:41.818727 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:41.818734 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:41.818747 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:41.818797 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:41.821112 (+ 2315us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 15:52:41.821337 (+ 225us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44792: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":263,"threads_started":1}
W20260929 15:52:41.821727 9668 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44792: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 15:52:41.821806 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 9208: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:38521: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 15:52:41.821991 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 9208
I20260929 15:52:41.838421 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 9359
I20260929 15:52:41.850203 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:41.839254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54818 (local address 127.0.97.2:36825)
0929 15:52:41.839825 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:41.839832 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:41.839862 (+ 30us) server_negotiation.cc:407] Connection header received
0929 15:52:41.839917 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:41.839924 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:41.840157 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:41.840369 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:41.841663 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:41.842695 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:41.843910 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:41.844447 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:41.844584 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:41.847149 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:41.847158 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:41.847171 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:41.847221 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:41.849525 (+ 2304us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 15:52:41.849830 (+ 305us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54818: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":290,"threads_started":1}
W20260929 15:52:41.850203 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 9359: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:36825: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 15:52:41.850301 9669 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54818: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 15:52:41.850338 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 9359
I20260929 15:52:41.866473 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 9490
I20260929 15:52:41.877696 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:41.867317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56224 (local address 127.0.97.3:39741)
0929 15:52:41.867879 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:41.867886 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:41.867911 (+ 25us) server_negotiation.cc:407] Connection header received
0929 15:52:41.867967 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:41.867973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:41.868181 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:41.868334 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:41.869618 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:41.870672 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:41.871767 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:41.872297 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:41.872393 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:41.874764 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:41.874772 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:41.874785 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:41.874819 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:41.877167 (+ 2348us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 15:52:41.877465 (+ 298us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56224: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":298,"threads_started":1}
W20260929 15:52:41.877815 9670 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56224: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 15:52:41.877805 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 9490: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:39741: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 15:52:41.877956 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 9490
I20260929 15:52:41.894223 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 9139
I20260929 15:52:41.895401 9201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:52:42.009658 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 9139
2026-09-29T15:52:42Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20909 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T15:52:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:52:42Z Disabled control of system clock
I20260929 15:52:42.085865 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44471
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44471
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 15:52:42.398151 9675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:52:42.398422 9675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:52:42.398557 9675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:52:42.405640 9675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:52:42.405809 9675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:52:42.405895 9675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:52:42.406002 9675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:52:42.406072 9675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:52:42.414875 9675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44471
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:52:42.416064 9675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:52:42.417785 9675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:52:42.429246 9680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:42.429282 9683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:42.429280 9681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:52:42.429565 9675 server_base.cc:1034] running on GCE node
I20260929 15:52:42.430461 9675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:52:42.431689 9675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:52:42.432886 9675 hybrid_clock.cc:648] HybridClock initialized: now 1790697162432850 us; error 24 us; skew 500 ppm
I20260929 15:52:42.433173 9675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:52:42.435314 9675 webserver.cc:460] Webserver started at http://127.0.97.62:33641/ using document root <none> and password file <none>
I20260929 15:52:42.435918 9675 fs_manager.cc:359] Metadata directory not provided
I20260929 15:52:42.435998 9675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:52:42.436249 9675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:52:42.438906 9675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "339db70a1d0449399c77f1ffeb550e29"
format_stamp: "Formatted at 2026-09-29 15:52:42 on dist-test-slave-r0ck"
server_key: "c8dd7d4951abf838494536fa244ed5f1"
server_key_iv: "3a95e05232d65ac6de2cfeed997047a0"
server_key_version: "encryptionkey@0"
I20260929 15:52:42.439584 9675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "339db70a1d0449399c77f1ffeb550e29"
format_stamp: "Formatted at 2026-09-29 15:52:42 on dist-test-slave-r0ck"
server_key: "c8dd7d4951abf838494536fa244ed5f1"
server_key_iv: "3a95e05232d65ac6de2cfeed997047a0"
server_key_version: "encryptionkey@0"
I20260929 15:52:42.444247 9675 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260929 15:52:42.447470 9689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:42.448725 9675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 15:52:42.448920 9675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "339db70a1d0449399c77f1ffeb550e29"
format_stamp: "Formatted at 2026-09-29 15:52:42 on dist-test-slave-r0ck"
server_key: "c8dd7d4951abf838494536fa244ed5f1"
server_key_iv: "3a95e05232d65ac6de2cfeed997047a0"
server_key_version: "encryptionkey@0"
I20260929 15:52:42.449113 9675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:52:42.470038 9675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:52:42.470799 9675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:52:42.471076 9675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:52:42.488797 9675 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44471
I20260929 15:52:42.488833 9740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44471 every 8 connection(s)
I20260929 15:52:42.490103 9675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:52:42.493762 9741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:52:42.495175 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 9675
I20260929 15:52:42.495355 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:52:42.495966 388 external_mini_cluster.cc:1428] Setting key e2f757637b81d212636f1cd00e64ffdb
I20260929 15:52:42.506600 9741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 339db70a1d0449399c77f1ffeb550e29: Bootstrap starting.
I20260929 15:52:42.510272 9741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 339db70a1d0449399c77f1ffeb550e29: Neither blocks nor log segments found. Creating new log.
I20260929 15:52:42.511196 9744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:42.497900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44112 (local address 127.0.97.62:44471)
0929 15:52:42.498937 (+ 1037us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:42.498953 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:42.499002 (+ 49us) server_negotiation.cc:407] Connection header received
0929 15:52:42.501237 (+ 2235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:42.501265 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:42.501816 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:42.502232 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:42.502748 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:42.504777 (+ 2029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:42.506088 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:42.507094 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:42.507281 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:42.508345 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:42.508354 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:42.508374 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:52:42.509059 (+ 685us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:42.509183 (+ 124us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:52:42.509208 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:42.509333 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:42.509566 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 15:52:42.509853 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":311,"threads_started":1}
I20260929 15:52:42.511526 9741 log.cc:824] T 00000000000000000000000000000000 P 339db70a1d0449399c77f1ffeb550e29: Log is configured to *not* fsync() on all Append() calls
I20260929 15:52:42.514503 9741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 339db70a1d0449399c77f1ffeb550e29: No bootstrap required, opened a new log
I20260929 15:52:42.521330 9741 raft_consensus.cc:348] T 00000000000000000000000000000000 P 339db70a1d0449399c77f1ffeb550e29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "339db70a1d0449399c77f1ffeb550e29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44471 } }
I20260929 15:52:42.521710 9741 raft_consensus.cc:374] T 00000000000000000000000000000000 P 339db70a1d0449399c77f1ffeb550e29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:52:42.521823 9741 raft_consensus.cc:729] T 00000000000000000000000000000000 P 339db70a1d0449399c77f1ffeb550e29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 339db70a1d0449399c77f1ffeb550e29, State: Initialized, Role: FOLLOWER
I20260929 15:52:42.522388 9741 consensus_queue.cc:260] T 00000000000000000000000000000000 P 339db70a1d0449399c77f1ffeb550e29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "339db70a1d0449399c77f1ffeb550e29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44471 } }
I20260929 15:52:42.522596 9741 raft_consensus.cc:388] T 00000000000000000000000000000000 P 339db70a1d0449399c77f1ffeb550e29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:52:42.522749 9741 raft_consensus.cc:482] T 00000000000000000000000000000000 P 339db70a1d0449399c77f1ffeb550e29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:52:42.522904 9741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 339db70a1d0449399c77f1ffeb550e29 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:52:42.525035 9741 raft_consensus.cc:504] T 00000000000000000000000000000000 P 339db70a1d0449399c77f1ffeb550e29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "339db70a1d0449399c77f1ffeb550e29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44471 } }
I20260929 15:52:42.525544 9741 leader_election.cc:302] T 00000000000000000000000000000000 P 339db70a1d0449399c77f1ffeb550e29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 339db70a1d0449399c77f1ffeb550e29; no voters:
I20260929 15:52:42.526051 9741 leader_election.cc:288] T 00000000000000000000000000000000 P 339db70a1d0449399c77f1ffeb550e29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:52:42.526183 9746 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 339db70a1d0449399c77f1ffeb550e29 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:52:42.526834 9746 raft_consensus.cc:686] T 00000000000000000000000000000000 P 339db70a1d0449399c77f1ffeb550e29 [term 1 LEADER]: Becoming Leader. State: Replica: 339db70a1d0449399c77f1ffeb550e29, State: Running, Role: LEADER
I20260929 15:52:42.527280 9746 consensus_queue.cc:237] T 00000000000000000000000000000000 P 339db70a1d0449399c77f1ffeb550e29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "339db70a1d0449399c77f1ffeb550e29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44471 } }
I20260929 15:52:42.527918 9741 sys_catalog.cc:564] T 00000000000000000000000000000000 P 339db70a1d0449399c77f1ffeb550e29 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:52:42.529971 9748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 339db70a1d0449399c77f1ffeb550e29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 339db70a1d0449399c77f1ffeb550e29. Latest consensus state: current_term: 1 leader_uuid: "339db70a1d0449399c77f1ffeb550e29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "339db70a1d0449399c77f1ffeb550e29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44471 } } }
I20260929 15:52:42.530382 9748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 339db70a1d0449399c77f1ffeb550e29 [sys.catalog]: This master's current role is: LEADER
I20260929 15:52:42.530896 9747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 339db70a1d0449399c77f1ffeb550e29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "339db70a1d0449399c77f1ffeb550e29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "339db70a1d0449399c77f1ffeb550e29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44471 } } }
I20260929 15:52:42.531247 9747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 339db70a1d0449399c77f1ffeb550e29 [sys.catalog]: This master's current role is: LEADER
I20260929 15:52:42.532647 9755 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:52:42.537330 9755 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:52:42.545732 9755 catalog_manager.cc:1347] Generated new cluster ID: a98c2719448648d1b2429f1a493f7133
I20260929 15:52:42.545853 9755 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:52:42.571985 9755 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:52:42.573163 9755 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:52:42.584120 9755 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 339db70a1d0449399c77f1ffeb550e29: Generated new TSK 0
I20260929 15:52:42.585137 9755 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:52:42.594693 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44471
--builtin_ntp_servers=127.0.97.20:35673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 15:52:42.919528 9765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:52:42.919780 9765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:52:42.919862 9765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:52:42.926942 9765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:52:42.927029 9765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:52:42.927126 9765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:52:42.935732 9765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:52:42.936865 9765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:52:42.938452 9765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:52:42.950695 9770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:42.950773 9771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:42.951071 9773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:52:42.951927 9765 server_base.cc:1034] running on GCE node
I20260929 15:52:42.952674 9765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:52:42.953715 9765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:52:42.954945 9765 hybrid_clock.cc:648] HybridClock initialized: now 1790697162954885 us; error 51 us; skew 500 ppm
I20260929 15:52:42.955282 9765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:52:42.965153 9765 webserver.cc:460] Webserver started at http://127.0.97.1:32871/ using document root <none> and password file <none>
I20260929 15:52:42.965917 9765 fs_manager.cc:359] Metadata directory not provided
I20260929 15:52:42.966032 9765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:52:42.966363 9765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:52:42.969113 9765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-0/data/instance:
uuid: "0d02c1fcb0ec448bb0051570cad6c142"
format_stamp: "Formatted at 2026-09-29 15:52:42 on dist-test-slave-r0ck"
server_key: "80f01a9ebd0df39dcd6701d495ee2533"
server_key_iv: "87c8d499fcc09aad27fea983b918d24a"
server_key_version: "encryptionkey@0"
I20260929 15:52:42.969942 9765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "0d02c1fcb0ec448bb0051570cad6c142"
format_stamp: "Formatted at 2026-09-29 15:52:42 on dist-test-slave-r0ck"
server_key: "80f01a9ebd0df39dcd6701d495ee2533"
server_key_iv: "87c8d499fcc09aad27fea983b918d24a"
server_key_version: "encryptionkey@0"
I20260929 15:52:42.974881 9765 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 15:52:42.978147 9779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:42.979410 9765 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:52:42.979545 9765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-0/wal
uuid: "0d02c1fcb0ec448bb0051570cad6c142"
format_stamp: "Formatted at 2026-09-29 15:52:42 on dist-test-slave-r0ck"
server_key: "80f01a9ebd0df39dcd6701d495ee2533"
server_key_iv: "87c8d499fcc09aad27fea983b918d24a"
server_key_version: "encryptionkey@0"
I20260929 15:52:42.979714 9765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:52:43.003055 9765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:52:43.003840 9765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:52:43.004092 9765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:52:43.005043 9765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:52:43.006628 9765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:52:43.006872 9765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:43.006978 9765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:52:43.007041 9765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:43.040822 9765 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39225
I20260929 15:52:43.040910 9891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39225 every 8 connection(s)
I20260929 15:52:43.042169 9765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:52:43.049274 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 9765
I20260929 15:52:43.049504 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:52:43.050139 388 external_mini_cluster.cc:1428] Setting key aada30b49727d9b7e74d2bfebfc40f19
I20260929 15:52:43.052845 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44471
--builtin_ntp_servers=127.0.97.20:35673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 15:52:43.057950 9894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:43.044887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43897 (local address 127.0.97.62:44471)
0929 15:52:43.045330 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:43.045336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:43.047067 (+ 1731us) server_negotiation.cc:407] Connection header received
0929 15:52:43.049795 (+ 2728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:43.049802 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:43.049912 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:43.050041 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:43.051727 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:43.052798 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:43.054982 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:43.055442 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:43.055588 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:43.057055 (+ 1467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:43.057060 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:43.057064 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:52:43.057081 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:43.057120 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:52:43.057131 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:43.057227 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:43.057512 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 15:52:43.057613 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":209,"threads_started":1}
I20260929 15:52:43.059087 9895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:43.045498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.1:43897)
0929 15:52:43.046799 (+ 1301us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:43.046860 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:43.049317 (+ 2457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:43.050385 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:43.050395 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:43.050521 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:43.051506 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:43.051534 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:43.052980 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:43.052987 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:43.054734 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:43.054759 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:43.055686 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:43.055696 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:43.055961 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:43.056702 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:52:43.056704 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:43.056749 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:52:43.056841 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:43.057226 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:43.057238 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:43.057249 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 15:52:43.057492 (+ 243us) client_negotiation.cc:240] Negotiation successful
0929 15:52:43.057953 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":305,"threads_started":1}
I20260929 15:52:43.060628 9892 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44471
I20260929 15:52:43.060957 9892 heartbeater.cc:464] Registering TS with master...
I20260929 15:52:43.061899 9892 heartbeater.cc:511] Master 127.0.97.62:44471 requested a full tablet report, sending...
I20260929 15:52:43.064245 9706 ts_manager.cc:194] Registered new tserver with Master: 0d02c1fcb0ec448bb0051570cad6c142 (127.0.97.1:39225)
I20260929 15:52:43.066072 9706 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:43897
W20260929 15:52:43.386319 9896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:52:43.386628 9896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:52:43.386797 9896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:52:43.393944 9896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:52:43.394033 9896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:52:43.394133 9896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:52:43.403082 9896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:52:43.404289 9896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:52:43.405967 9896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:52:43.418812 9901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:43.418748 9904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:43.418853 9902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:52:43.419685 9896 server_base.cc:1034] running on GCE node
I20260929 15:52:43.420264 9896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:52:43.421161 9896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:52:43.422421 9896 hybrid_clock.cc:648] HybridClock initialized: now 1790697163422368 us; error 43 us; skew 500 ppm
I20260929 15:52:43.422778 9896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:52:43.425101 9896 webserver.cc:460] Webserver started at http://127.0.97.2:33811/ using document root <none> and password file <none>
I20260929 15:52:43.425832 9896 fs_manager.cc:359] Metadata directory not provided
I20260929 15:52:43.425933 9896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:52:43.426191 9896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:52:43.429031 9896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-1/data/instance:
uuid: "a54df5cc7e634b65865da633f587e12f"
format_stamp: "Formatted at 2026-09-29 15:52:43 on dist-test-slave-r0ck"
server_key: "f8f2c74b10f14f891b36c4d55291b348"
server_key_iv: "00a7a458e3d3f65618fe98c674d31437"
server_key_version: "encryptionkey@0"
I20260929 15:52:43.429809 9896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a54df5cc7e634b65865da633f587e12f"
format_stamp: "Formatted at 2026-09-29 15:52:43 on dist-test-slave-r0ck"
server_key: "f8f2c74b10f14f891b36c4d55291b348"
server_key_iv: "00a7a458e3d3f65618fe98c674d31437"
server_key_version: "encryptionkey@0"
I20260929 15:52:43.434955 9896 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 15:52:43.438201 9910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:43.439461 9896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:52:43.439589 9896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-1/wal
uuid: "a54df5cc7e634b65865da633f587e12f"
format_stamp: "Formatted at 2026-09-29 15:52:43 on dist-test-slave-r0ck"
server_key: "f8f2c74b10f14f891b36c4d55291b348"
server_key_iv: "00a7a458e3d3f65618fe98c674d31437"
server_key_version: "encryptionkey@0"
I20260929 15:52:43.439760 9896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:52:43.460160 9896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:52:43.460912 9896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:52:43.461136 9896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:52:43.461997 9896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:52:43.463358 9896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:52:43.463440 9896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:43.463507 9896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:52:43.463552 9896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:43.496662 9896 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33617
I20260929 15:52:43.496724 10022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33617 every 8 connection(s)
I20260929 15:52:43.497993 9896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:52:43.498839 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 9896
I20260929 15:52:43.499023 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:52:43.499704 388 external_mini_cluster.cc:1428] Setting key d2d8ed613adb65a3311ceeff78bb9962
I20260929 15:52:43.502852 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44471
--builtin_ntp_servers=127.0.97.20:35673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 15:52:43.513453 9894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:43.500854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56197 (local address 127.0.97.62:44471)
0929 15:52:43.501032 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:43.501038 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:43.502711 (+ 1673us) server_negotiation.cc:407] Connection header received
0929 15:52:43.505448 (+ 2737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:43.505454 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:43.505530 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:43.505623 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:43.506890 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:43.507618 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:43.510169 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:43.510904 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:43.511180 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:43.512516 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:43.512521 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:43.512524 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:52:43.512541 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:43.512571 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:52:43.512593 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:43.512664 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:43.512962 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 15:52:43.513069 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 15:52:43.515165 10025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:43.501325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.2:56197)
0929 15:52:43.502539 (+ 1214us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:43.502592 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:43.505142 (+ 2550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:43.505852 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:43.505865 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:43.505974 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:43.506716 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:43.506740 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:43.507761 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:43.507769 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:43.509933 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:43.509951 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:43.511227 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:43.511232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:43.511430 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:43.512141 (+ 711us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:52:43.512147 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:43.512196 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:52:43.512298 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:43.512690 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:43.512702 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:43.512713 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 15:52:43.512896 (+ 183us) client_negotiation.cc:240] Negotiation successful
0929 15:52:43.513305 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":428,"threads_started":1}
I20260929 15:52:43.517096 10023 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44471
I20260929 15:52:43.517490 10023 heartbeater.cc:464] Registering TS with master...
I20260929 15:52:43.518245 10023 heartbeater.cc:511] Master 127.0.97.62:44471 requested a full tablet report, sending...
I20260929 15:52:43.519737 9706 ts_manager.cc:194] Registered new tserver with Master: a54df5cc7e634b65865da633f587e12f (127.0.97.2:33617)
I20260929 15:52:43.520941 9706 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:56197
W20260929 15:52:43.848824 10026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:52:43.849112 10026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:52:43.849203 10026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:52:43.857043 10026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:52:43.857138 10026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:52:43.857240 10026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:52:43.866869 10026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:52:43.868028 10026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:52:43.869771 10026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:52:43.882932 10032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:43.883049 10034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:43.883352 10031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:52:43.884485 10026 server_base.cc:1034] running on GCE node
I20260929 15:52:43.885067 10026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:52:43.886174 10026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:52:43.887594 10026 hybrid_clock.cc:648] HybridClock initialized: now 1790697163887469 us; error 113 us; skew 500 ppm
I20260929 15:52:43.887912 10026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:52:43.891021 10026 webserver.cc:460] Webserver started at http://127.0.97.3:39743/ using document root <none> and password file <none>
I20260929 15:52:43.891757 10026 fs_manager.cc:359] Metadata directory not provided
I20260929 15:52:43.891857 10026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:52:43.892213 10026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:52:43.895226 10026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-2/data/instance:
uuid: "ad2b76fc864d47df8f58eab0e8d8f86f"
format_stamp: "Formatted at 2026-09-29 15:52:43 on dist-test-slave-r0ck"
server_key: "46fdeeae5e8fc019d949c16c8507e6a5"
server_key_iv: "96bcbb301c132545c90b3b2b68ad52d9"
server_key_version: "encryptionkey@0"
I20260929 15:52:43.896055 10026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ad2b76fc864d47df8f58eab0e8d8f86f"
format_stamp: "Formatted at 2026-09-29 15:52:43 on dist-test-slave-r0ck"
server_key: "46fdeeae5e8fc019d949c16c8507e6a5"
server_key_iv: "96bcbb301c132545c90b3b2b68ad52d9"
server_key_version: "encryptionkey@0"
I20260929 15:52:43.901443 10026 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 15:52:43.904969 10040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:43.906373 10026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:52:43.906540 10026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-2/wal
uuid: "ad2b76fc864d47df8f58eab0e8d8f86f"
format_stamp: "Formatted at 2026-09-29 15:52:43 on dist-test-slave-r0ck"
server_key: "46fdeeae5e8fc019d949c16c8507e6a5"
server_key_iv: "96bcbb301c132545c90b3b2b68ad52d9"
server_key_version: "encryptionkey@0"
I20260929 15:52:43.906751 10026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:52:43.923784 10026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:52:43.924647 10026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:52:43.924943 10026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:52:43.925935 10026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:52:43.927597 10026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:52:43.927723 10026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:43.927865 10026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:52:43.927959 10026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:43.970245 10026 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42167
I20260929 15:52:43.970288 10152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42167 every 8 connection(s)
I20260929 15:52:43.971535 10026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:52:43.972182 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 10026
I20260929 15:52:43.972476 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697075657932-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:52:43.973265 388 external_mini_cluster.cc:1428] Setting key 6cd7c48474a5ea33f363eb46af2dcc8f
I20260929 15:52:43.985337 9894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:43.974320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57481 (local address 127.0.97.62:44471)
0929 15:52:43.974516 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:43.974522 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:43.976377 (+ 1855us) server_negotiation.cc:407] Connection header received
0929 15:52:43.978191 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:43.978195 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:43.978258 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:43.978346 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:43.979422 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:43.980282 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:43.982295 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:43.982783 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:43.982896 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:43.984626 (+ 1730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:43.984632 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:43.984635 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:52:43.984655 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:43.984694 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:52:43.984706 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:43.984789 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:43.984995 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 15:52:43.985086 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 15:52:43.986522 10155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:43.974929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.3:57481)
0929 15:52:43.976227 (+ 1298us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:43.976274 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:43.977941 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:43.978560 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:43.978567 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:43.978645 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:43.979190 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:43.979205 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:43.980527 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:43.980533 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:43.982115 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:43.982133 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:43.983032 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:43.983039 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:43.983377 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:43.984236 (+ 859us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:52:43.984239 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:43.984290 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:52:43.984360 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:43.984813 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:43.984821 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:43.984826 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 15:52:43.984964 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 15:52:43.985257 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":346,"threads_started":1}
I20260929 15:52:43.988161 10153 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44471
I20260929 15:52:43.988514 10153 heartbeater.cc:464] Registering TS with master...
I20260929 15:52:43.989456 10153 heartbeater.cc:511] Master 127.0.97.62:44471 requested a full tablet report, sending...
I20260929 15:52:43.990772 9706 ts_manager.cc:194] Registered new tserver with Master: ad2b76fc864d47df8f58eab0e8d8f86f (127.0.97.3:42167)
I20260929 15:52:43.991582 9706 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:57481
I20260929 15:52:44.001493 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 15:52:44.008294 9894 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:44.006702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44128 (local address 127.0.97.62:44471)
0929 15:52:44.007005 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:44.007012 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:44.007299 (+ 287us) server_negotiation.cc:407] Connection header received
0929 15:52:44.007504 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:44.007513 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:44.007584 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:44.007686 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:44.008105 (+ 419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44128: BlockingRecv error: recv got EOF from 127.0.0.1:44128 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
I20260929 15:52:44.009485 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 9765
I20260929 15:52:44.019908 10162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:44.011517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48660 (local address 127.0.97.1:39225)
0929 15:52:44.012139 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:44.012148 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:44.012183 (+ 35us) server_negotiation.cc:407] Connection header received
0929 15:52:44.012260 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:44.012268 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:44.012534 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:44.012704 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:44.013204 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:44.014628 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:44.016122 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:44.016693 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:44.016814 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:44.018083 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:44.018100 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:44.018118 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:52:44.018899 (+ 781us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:44.019007 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:52:44.019032 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:44.019174 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:44.019403 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 15:52:44.019556 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":306,"threads_started":1}
I20260929 15:52:44.021544 9887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:52:44.107920 9892 heartbeater.cc:503] Master 127.0.97.62:44471 was elected leader, sending a full tablet report...
I20260929 15:52:44.109707 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 9765
I20260929 15:52:44.126437 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 9896
I20260929 15:52:44.135239 10164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:44.127333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45276 (local address 127.0.97.2:33617)
0929 15:52:44.127878 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:44.127885 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:44.127919 (+ 34us) server_negotiation.cc:407] Connection header received
0929 15:52:44.127976 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:44.127982 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:44.128220 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:44.128429 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:44.128842 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:44.129948 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:44.131240 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:44.132179 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:44.132314 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:44.133299 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:44.133307 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:44.133324 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:52:44.134053 (+ 729us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:44.134098 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:52:44.134117 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:44.134213 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:44.134680 (+ 467us) server_negotiation.cc:299] Negotiation successful
0929 15:52:44.134843 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":267,"threads_started":1}
I20260929 15:52:44.136693 10018 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:52:44.223678 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 9896
I20260929 15:52:44.241011 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 10026
I20260929 15:52:44.249752 10166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:44.242024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48906 (local address 127.0.97.3:42167)
0929 15:52:44.242671 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:44.242678 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:44.242710 (+ 32us) server_negotiation.cc:407] Connection header received
0929 15:52:44.242774 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:44.242780 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:44.243008 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:44.243187 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:44.243659 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:44.244757 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:44.246205 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:44.246786 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:44.246939 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:44.248326 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:44.248335 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:44.248348 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:52:44.248908 (+ 560us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:44.248957 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:52:44.248976 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:44.249101 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:44.249256 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 15:52:44.249427 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":350,"threads_started":1}
I20260929 15:52:44.250941 10148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:52:44.335484 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 10026
I20260929 15:52:44.352347 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 9675
I20260929 15:52:44.353657 9736 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:52:44.454615 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 9675
2026-09-29T15:52:44Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2441 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T15:52:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:52:44Z Disabled control of system clock
I20260929 15:52:44.533537 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41295
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 15:52:44.854193 10172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:52:44.854471 10172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:52:44.854600 10172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:52:44.861918 10172 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:52:44.862031 10172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:52:44.862069 10172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:52:44.862118 10172 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:52:44.862219 10172 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:52:44.872313 10172 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41295
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:52:44.874112 10172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:52:44.876349 10172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:52:44.889571 10177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:44.889865 10178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:44.889901 10180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:52:44.890496 10172 server_base.cc:1034] running on GCE node
I20260929 15:52:44.891307 10172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:52:44.892742 10172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:52:44.893968 10172 hybrid_clock.cc:648] HybridClock initialized: now 1790697164893929 us; error 62 us; skew 500 ppm
I20260929 15:52:44.894306 10172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:52:44.896642 10172 webserver.cc:460] Webserver started at http://127.0.97.62:35349/ using document root <none> and password file <none>
I20260929 15:52:44.897320 10172 fs_manager.cc:359] Metadata directory not provided
I20260929 15:52:44.897476 10172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:52:44.897805 10172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:52:44.900835 10172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "0c2be2c15d38452088cae80353a6c623"
format_stamp: "Formatted at 2026-09-29 15:52:44 on dist-test-slave-r0ck"
server_key: "1c75c64dbd13fe8c403f88639e00fe21"
server_key_iv: "ad142b6034a675d07941e651cb500551"
server_key_version: "encryptionkey@0"
I20260929 15:52:44.901684 10172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "0c2be2c15d38452088cae80353a6c623"
format_stamp: "Formatted at 2026-09-29 15:52:44 on dist-test-slave-r0ck"
server_key: "1c75c64dbd13fe8c403f88639e00fe21"
server_key_iv: "ad142b6034a675d07941e651cb500551"
server_key_version: "encryptionkey@0"
I20260929 15:52:44.906646 10172 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 15:52:44.910590 10186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:44.911897 10172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:52:44.912106 10172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "0c2be2c15d38452088cae80353a6c623"
format_stamp: "Formatted at 2026-09-29 15:52:44 on dist-test-slave-r0ck"
server_key: "1c75c64dbd13fe8c403f88639e00fe21"
server_key_iv: "ad142b6034a675d07941e651cb500551"
server_key_version: "encryptionkey@0"
I20260929 15:52:44.912343 10172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:52:44.919126 10172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:52:44.920189 10172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:52:44.920555 10172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:52:44.940267 10172 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41295
I20260929 15:52:44.940285 10237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41295 every 8 connection(s)
I20260929 15:52:44.941591 10172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:52:44.945204 10238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:52:44.945410 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 10172
I20260929 15:52:44.945628 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:52:44.946451 388 external_mini_cluster.cc:1428] Setting key 365fec679739d4a66a15a249b42ad40b
I20260929 15:52:44.955291 10241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:44.948924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35608 (local address 127.0.97.62:41295)
0929 15:52:44.950312 (+ 1388us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:44.950323 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:44.950365 (+ 42us) server_negotiation.cc:407] Connection header received
0929 15:52:44.951992 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:44.952013 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:44.952408 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:44.952735 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:44.953797 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:44.953804 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:44.953815 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:52:44.953831 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:44.953892 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:52:44.953910 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:44.954095 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:44.954192 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 15:52:44.954476 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":932,"thread_start_us":659,"threads_started":1}
I20260929 15:52:44.958550 10238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c2be2c15d38452088cae80353a6c623: Bootstrap starting.
I20260929 15:52:44.961229 10238 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c2be2c15d38452088cae80353a6c623: Neither blocks nor log segments found. Creating new log.
I20260929 15:52:44.962230 10238 log.cc:824] T 00000000000000000000000000000000 P 0c2be2c15d38452088cae80353a6c623: Log is configured to *not* fsync() on all Append() calls
I20260929 15:52:44.965310 10238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c2be2c15d38452088cae80353a6c623: No bootstrap required, opened a new log
I20260929 15:52:44.972699 10238 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c2be2c15d38452088cae80353a6c623 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c2be2c15d38452088cae80353a6c623" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41295 } }
I20260929 15:52:44.973006 10238 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c2be2c15d38452088cae80353a6c623 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:52:44.973132 10238 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c2be2c15d38452088cae80353a6c623 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c2be2c15d38452088cae80353a6c623, State: Initialized, Role: FOLLOWER
I20260929 15:52:44.973781 10238 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c2be2c15d38452088cae80353a6c623 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c2be2c15d38452088cae80353a6c623" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41295 } }
I20260929 15:52:44.974000 10238 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c2be2c15d38452088cae80353a6c623 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:52:44.974145 10238 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c2be2c15d38452088cae80353a6c623 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:52:44.974310 10238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c2be2c15d38452088cae80353a6c623 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:52:44.976310 10238 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c2be2c15d38452088cae80353a6c623 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c2be2c15d38452088cae80353a6c623" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41295 } }
I20260929 15:52:44.976776 10238 leader_election.cc:302] T 00000000000000000000000000000000 P 0c2be2c15d38452088cae80353a6c623 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c2be2c15d38452088cae80353a6c623; no voters:
I20260929 15:52:44.977507 10238 leader_election.cc:288] T 00000000000000000000000000000000 P 0c2be2c15d38452088cae80353a6c623 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:52:44.977722 10243 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c2be2c15d38452088cae80353a6c623 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:52:44.978947 10243 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c2be2c15d38452088cae80353a6c623 [term 1 LEADER]: Becoming Leader. State: Replica: 0c2be2c15d38452088cae80353a6c623, State: Running, Role: LEADER
I20260929 15:52:44.979526 10243 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c2be2c15d38452088cae80353a6c623 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c2be2c15d38452088cae80353a6c623" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41295 } }
I20260929 15:52:44.979646 10238 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c2be2c15d38452088cae80353a6c623 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:52:44.982481 10244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c2be2c15d38452088cae80353a6c623 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c2be2c15d38452088cae80353a6c623" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c2be2c15d38452088cae80353a6c623" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41295 } } }
I20260929 15:52:44.983744 10245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c2be2c15d38452088cae80353a6c623 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c2be2c15d38452088cae80353a6c623. Latest consensus state: current_term: 1 leader_uuid: "0c2be2c15d38452088cae80353a6c623" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c2be2c15d38452088cae80353a6c623" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41295 } } }
I20260929 15:52:44.984200 10245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c2be2c15d38452088cae80353a6c623 [sys.catalog]: This master's current role is: LEADER
I20260929 15:52:44.984200 10244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c2be2c15d38452088cae80353a6c623 [sys.catalog]: This master's current role is: LEADER
I20260929 15:52:44.985119 10252 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:52:44.992556 10252 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:52:45.001228 10252 catalog_manager.cc:1347] Generated new cluster ID: 5aa7f43e0b0f49b787b2ee631d6df291
I20260929 15:52:45.001361 10252 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:52:45.052649 10252 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:52:45.053123 10252 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:52:45.060108 10252 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c2be2c15d38452088cae80353a6c623: Generated new TSK 0
I20260929 15:52:45.061056 10252 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:52:45.072376 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41295
--builtin_ntp_servers=127.0.97.20:45645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 15:52:45.397204 10262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:52:45.397519 10262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:52:45.397614 10262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:52:45.404966 10262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:52:45.405056 10262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:52:45.405159 10262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:52:45.414059 10262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:52:45.415251 10262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:52:45.417049 10262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:52:45.429864 10267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:45.430181 10270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:45.430332 10268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:52:45.431181 10262 server_base.cc:1034] running on GCE node
I20260929 15:52:45.431751 10262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:52:45.432662 10262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:52:45.433969 10262 hybrid_clock.cc:648] HybridClock initialized: now 1790697165433890 us; error 39 us; skew 500 ppm
I20260929 15:52:45.434520 10262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:52:45.437162 10262 webserver.cc:460] Webserver started at http://127.0.97.1:42427/ using document root <none> and password file <none>
I20260929 15:52:45.437858 10262 fs_manager.cc:359] Metadata directory not provided
I20260929 15:52:45.437944 10262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:52:45.438272 10262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:52:45.443580 10262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-0/data/instance:
uuid: "2ee442819f9a4115a4e3b8b27f43cabf"
format_stamp: "Formatted at 2026-09-29 15:52:45 on dist-test-slave-r0ck"
server_key: "099cf9131718781ca1a807862cf68cfa"
server_key_iv: "2a26f46b700975d349135ac4d1bba390"
server_key_version: "encryptionkey@0"
I20260929 15:52:45.444448 10262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "2ee442819f9a4115a4e3b8b27f43cabf"
format_stamp: "Formatted at 2026-09-29 15:52:45 on dist-test-slave-r0ck"
server_key: "099cf9131718781ca1a807862cf68cfa"
server_key_iv: "2a26f46b700975d349135ac4d1bba390"
server_key_version: "encryptionkey@0"
I20260929 15:52:45.449334 10262 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 15:52:45.452669 10276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:45.454046 10262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:52:45.454211 10262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-0/wal
uuid: "2ee442819f9a4115a4e3b8b27f43cabf"
format_stamp: "Formatted at 2026-09-29 15:52:45 on dist-test-slave-r0ck"
server_key: "099cf9131718781ca1a807862cf68cfa"
server_key_iv: "2a26f46b700975d349135ac4d1bba390"
server_key_version: "encryptionkey@0"
I20260929 15:52:45.454435 10262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:52:45.459920 10262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:52:45.460650 10262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:52:45.460912 10262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:52:45.461851 10262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:52:45.463294 10262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:52:45.463408 10262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:45.463532 10262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:52:45.463616 10262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:45.498525 10262 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43065
I20260929 15:52:45.498575 10388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43065 every 8 connection(s)
I20260929 15:52:45.499830 10262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:52:45.504021 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 10262
I20260929 15:52:45.504226 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:52:45.504841 388 external_mini_cluster.cc:1428] Setting key 23b6d3393d3252368b822dac06dca6d0
I20260929 15:52:45.508221 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41295
--builtin_ntp_servers=127.0.97.20:45645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 15:52:45.509619 10391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:45.502466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38869 (local address 127.0.97.62:41295)
0929 15:52:45.503028 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:45.503037 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:45.504093 (+ 1056us) server_negotiation.cc:407] Connection header received
0929 15:52:45.506442 (+ 2349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:45.506448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:45.506510 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:45.506642 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:45.508441 (+ 1799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:45.508561 (+ 120us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:45.508570 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:52:45.508601 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:45.508656 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:52:45.508692 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:45.508983 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:45.509232 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 15:52:45.509307 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":306,"threads_started":1}
I20260929 15:52:45.510555 10392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:45.502768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41295 (local address 127.0.97.1:38869)
0929 15:52:45.503866 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:45.503926 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:45.506067 (+ 2141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:45.506993 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:45.507007 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:45.507154 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:45.508059 (+ 905us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:52:45.508062 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:45.508112 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:52:45.508216 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:45.508885 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:45.508899 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:45.508977 (+ 78us) client_negotiation.cc:769] Sending connection context
0929 15:52:45.509330 (+ 353us) client_negotiation.cc:240] Negotiation successful
0929 15:52:45.509638 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":316,"threads_started":1}
I20260929 15:52:45.512122 10389 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41295
I20260929 15:52:45.512411 10389 heartbeater.cc:464] Registering TS with master...
I20260929 15:52:45.512977 10389 heartbeater.cc:511] Master 127.0.97.62:41295 requested a full tablet report, sending...
I20260929 15:52:45.515031 10203 ts_manager.cc:194] Registered new tserver with Master: 2ee442819f9a4115a4e3b8b27f43cabf (127.0.97.1:43065)
W20260929 15:52:45.842631 10393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:52:45.842900 10393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:52:45.843024 10393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:52:45.850222 10393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:52:45.850327 10393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:52:45.850485 10393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:52:45.859629 10393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:52:45.860890 10393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:52:45.862643 10393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:52:45.875113 10401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:45.875205 10398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:45.875141 10399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:52:45.875811 10393 server_base.cc:1034] running on GCE node
I20260929 15:52:45.876328 10393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:52:45.877177 10393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:52:45.878504 10393 hybrid_clock.cc:648] HybridClock initialized: now 1790697165878400 us; error 57 us; skew 500 ppm
I20260929 15:52:45.879017 10393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:52:45.882377 10393 webserver.cc:460] Webserver started at http://127.0.97.2:35705/ using document root <none> and password file <none>
I20260929 15:52:45.883237 10393 fs_manager.cc:359] Metadata directory not provided
I20260929 15:52:45.883342 10393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:52:45.883664 10393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:52:45.887392 10393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-1/data/instance:
uuid: "67ed1240fd3243f9ac734b4ff69119b5"
format_stamp: "Formatted at 2026-09-29 15:52:45 on dist-test-slave-r0ck"
server_key: "a243616de7da464ce9b6e372dfd6f418"
server_key_iv: "c97e9136975142eabe33337324311023"
server_key_version: "encryptionkey@0"
I20260929 15:52:45.888536 10393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "67ed1240fd3243f9ac734b4ff69119b5"
format_stamp: "Formatted at 2026-09-29 15:52:45 on dist-test-slave-r0ck"
server_key: "a243616de7da464ce9b6e372dfd6f418"
server_key_iv: "c97e9136975142eabe33337324311023"
server_key_version: "encryptionkey@0"
I20260929 15:52:45.894106 10393 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 15:52:45.898020 10407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:45.899899 10393 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 15:52:45.900075 10393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-1/wal
uuid: "67ed1240fd3243f9ac734b4ff69119b5"
format_stamp: "Formatted at 2026-09-29 15:52:45 on dist-test-slave-r0ck"
server_key: "a243616de7da464ce9b6e372dfd6f418"
server_key_iv: "c97e9136975142eabe33337324311023"
server_key_version: "encryptionkey@0"
I20260929 15:52:45.900370 10393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:52:45.907351 10393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:52:45.908102 10393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:52:45.908360 10393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:52:45.909241 10393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:52:45.910555 10393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:52:45.910673 10393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:45.910786 10393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:52:45.910876 10393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:45.941604 10393 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41589
I20260929 15:52:45.941681 10519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41589 every 8 connection(s)
I20260929 15:52:45.943145 10393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:52:45.950731 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 10393
I20260929 15:52:45.950927 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:52:45.951470 388 external_mini_cluster.cc:1428] Setting key 88694b47cdf06c66c39cc958f5fcde32
I20260929 15:52:45.952535 10391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:45.945790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45119 (local address 127.0.97.62:41295)
0929 15:52:45.946031 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:45.946037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:45.947668 (+ 1631us) server_negotiation.cc:407] Connection header received
0929 15:52:45.949744 (+ 2076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:45.949751 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:45.949831 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:45.949956 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:45.951600 (+ 1644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:45.951606 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:45.951610 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:52:45.951627 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:45.951660 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:52:45.951675 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:45.951793 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:45.952167 (+ 374us) server_negotiation.cc:299] Negotiation successful
0929 15:52:45.952240 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 15:52:45.954197 10522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:45.946209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41295 (local address 127.0.97.2:45119)
0929 15:52:45.947450 (+ 1241us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:45.947512 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:45.949429 (+ 1917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:45.950218 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:45.950228 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:45.950398 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:45.951215 (+ 817us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:52:45.951219 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:45.951272 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:52:45.951384 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:45.951788 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:45.951799 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:45.951807 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 15:52:45.952878 (+ 1071us) client_negotiation.cc:240] Negotiation successful
0929 15:52:45.953075 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":357,"threads_started":1}
I20260929 15:52:45.954625 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41295
--builtin_ntp_servers=127.0.97.20:45645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 15:52:45.955726 10520 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41295
I20260929 15:52:45.955966 10520 heartbeater.cc:464] Registering TS with master...
I20260929 15:52:45.956625 10520 heartbeater.cc:511] Master 127.0.97.62:41295 requested a full tablet report, sending...
I20260929 15:52:45.958318 10203 ts_manager.cc:194] Registered new tserver with Master: 67ed1240fd3243f9ac734b4ff69119b5 (127.0.97.2:41589)
W20260929 15:52:46.285477 10523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:52:46.285838 10523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:52:46.286005 10523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:52:46.293111 10523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:52:46.293248 10523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:52:46.293432 10523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:52:46.303148 10523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:52:46.304466 10523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:52:46.306280 10523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:52:46.319031 10529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:46.319002 10528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:46.319237 10531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:52:46.319721 10523 server_base.cc:1034] running on GCE node
I20260929 15:52:46.320334 10523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:52:46.321233 10523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:52:46.322428 10523 hybrid_clock.cc:648] HybridClock initialized: now 1790697166322381 us; error 43 us; skew 500 ppm
I20260929 15:52:46.322690 10523 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:52:46.324910 10523 webserver.cc:460] Webserver started at http://127.0.97.3:36025/ using document root <none> and password file <none>
I20260929 15:52:46.325595 10523 fs_manager.cc:359] Metadata directory not provided
I20260929 15:52:46.325680 10523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:52:46.325929 10523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:52:46.328630 10523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-2/data/instance:
uuid: "3823a654c9be45c8a53db27b8fd08465"
format_stamp: "Formatted at 2026-09-29 15:52:46 on dist-test-slave-r0ck"
server_key: "37a0022ce9e0256844c668935d9e8d6f"
server_key_iv: "4a8775fcc308c7e72c00ed2736e06fa8"
server_key_version: "encryptionkey@0"
I20260929 15:52:46.329391 10523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "3823a654c9be45c8a53db27b8fd08465"
format_stamp: "Formatted at 2026-09-29 15:52:46 on dist-test-slave-r0ck"
server_key: "37a0022ce9e0256844c668935d9e8d6f"
server_key_iv: "4a8775fcc308c7e72c00ed2736e06fa8"
server_key_version: "encryptionkey@0"
I20260929 15:52:46.334028 10523 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 15:52:46.337168 10537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:46.338615 10523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 15:52:46.338743 10523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-2/wal
uuid: "3823a654c9be45c8a53db27b8fd08465"
format_stamp: "Formatted at 2026-09-29 15:52:46 on dist-test-slave-r0ck"
server_key: "37a0022ce9e0256844c668935d9e8d6f"
server_key_iv: "4a8775fcc308c7e72c00ed2736e06fa8"
server_key_version: "encryptionkey@0"
I20260929 15:52:46.338933 10523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:52:46.344527 10523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:52:46.345225 10523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:52:46.345502 10523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:52:46.346393 10523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:52:46.347709 10523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:52:46.347800 10523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:46.347879 10523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:52:46.347934 10523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:46.380797 10523 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36841
I20260929 15:52:46.380920 10649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36841 every 8 connection(s)
I20260929 15:52:46.382129 10523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:52:46.387508 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 10523
I20260929 15:52:46.387682 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697075657932-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:52:46.388083 388 external_mini_cluster.cc:1428] Setting key 1d8a2806c3ca0f426eec42b977b4a745
I20260929 15:52:46.390372 10391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:46.384688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33875 (local address 127.0.97.62:41295)
0929 15:52:46.384891 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:46.384898 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:46.386383 (+ 1485us) server_negotiation.cc:407] Connection header received
0929 15:52:46.388033 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:46.388039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:46.388119 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:46.388223 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:46.389659 (+ 1436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:46.389663 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:46.389666 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:52:46.389679 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:46.389719 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:52:46.389731 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:46.389846 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:46.390032 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 15:52:46.390092 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 15:52:46.391299 10652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:46.385204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41295 (local address 127.0.97.3:33875)
0929 15:52:46.386239 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:46.386281 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:46.387792 (+ 1511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:46.388522 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:46.388533 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:46.388644 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:46.389305 (+ 661us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:52:46.389308 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:46.389379 (+ 71us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:52:46.389502 (+ 123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:46.389827 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:46.389837 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:46.389845 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 15:52:46.390108 (+ 263us) client_negotiation.cc:240] Negotiation successful
0929 15:52:46.390333 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":374,"threads_started":1}
I20260929 15:52:46.392416 10650 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41295
I20260929 15:52:46.392683 10650 heartbeater.cc:464] Registering TS with master...
I20260929 15:52:46.393201 10650 heartbeater.cc:511] Master 127.0.97.62:41295 requested a full tablet report, sending...
I20260929 15:52:46.394506 10203 ts_manager.cc:194] Registered new tserver with Master: 3823a654c9be45c8a53db27b8fd08465 (127.0.97.3:36841)
I20260929 15:52:46.402982 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 15:52:46.409379 10391 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:52:46.407758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35612 (local address 127.0.97.62:41295)
0929 15:52:46.407907 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:46.407911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:46.408425 (+ 514us) server_negotiation.cc:407] Connection header received
0929 15:52:46.408616 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:46.408620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:46.408678 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:46.408783 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:46.409076 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35612: BlockingRecv error: recv got EOF from 127.0.0.1:35612 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 15:52:46.410216 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 10262
I20260929 15:52:46.413457 10659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:46.411130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55840 (local address 127.0.97.1:43065)
0929 15:52:46.411627 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:46.411634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:46.411658 (+ 24us) server_negotiation.cc:407] Connection header received
0929 15:52:46.411713 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:46.411719 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:46.411920 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:46.412040 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:46.412854 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:46.412860 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:46.412870 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:52:46.412886 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:46.412923 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:52:46.412936 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:46.413035 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:46.413117 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 15:52:46.413188 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":242,"threads_started":1}
I20260929 15:52:46.414548 10384 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:52:46.493333 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 10262
I20260929 15:52:46.510213 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 10393
I20260929 15:52:46.513739 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:46.511182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60390 (local address 127.0.97.2:41589)
0929 15:52:46.511645 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:46.511652 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:46.511676 (+ 24us) server_negotiation.cc:407] Connection header received
0929 15:52:46.511736 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:46.511742 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:46.511963 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:46.512127 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:46.513064 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:46.513070 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:46.513080 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:52:46.513092 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:46.513129 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:52:46.513141 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:46.513281 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:46.513395 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 15:52:46.513491 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":226,"threads_started":1}
I20260929 15:52:46.514873 10515 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:52:46.594975 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 10393
I20260929 15:52:46.611790 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 10523
I20260929 15:52:46.615646 10663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:46.612780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34958 (local address 127.0.97.3:36841)
0929 15:52:46.613383 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:46.613396 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:46.613425 (+ 29us) server_negotiation.cc:407] Connection header received
0929 15:52:46.613483 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:46.613490 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:46.613697 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:46.613854 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:46.614928 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:46.614934 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:46.614944 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:52:46.614959 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:46.614995 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:52:46.615008 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:46.615182 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:46.615262 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 15:52:46.615356 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":282,"threads_started":1}
I20260929 15:52:46.616860 10645 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:52:46.696888 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 10523
I20260929 15:52:46.712352 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 10172
I20260929 15:52:46.713501 10233 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:52:46.813221 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 10172
2026-09-29T15:52:46Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2356 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:52:46 dist-test-slave-r0ck krb5kdc[10668](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:52:46 dist-test-slave-r0ck krb5kdc[10668](info): set up 2 sockets
Sep 29 15:52:46 dist-test-slave-r0ck krb5kdc[10668](info): commencing operation
krb5kdc: starting...
W20260929 15:52:47.922667 388 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:52:48 dist-test-slave-r0ck krb5kdc[10668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:52:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:52:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:52:48.161701 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41961
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41961
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:52:48.477567 10681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:52:48.477892 10681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:52:48.478044 10681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:52:48.485108 10681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:52:48.485244 10681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:52:48.485378 10681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:52:48.485472 10681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:52:48.485553 10681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:52:48.495676 10681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:52:48.496914 10681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:52:48.498185 10681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:52:48.510479 10686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:48.510586 10687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:48.510569 10689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:52:48.511387 10681 server_base.cc:1034] running on GCE node
I20260929 15:52:48.512152 10681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:52:48.513592 10681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:52:48.514812 10681 hybrid_clock.cc:648] HybridClock initialized: now 1790697168514760 us; error 60 us; skew 500 ppm
I20260929 15:52:48.515144 10681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:52:48 dist-test-slave-r0ck krb5kdc[10668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:52:48.521193 10681 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:52:48.522658 10681 webserver.cc:460] Webserver started at http://127.0.97.62:33643/ using document root <none> and password file <none>
I20260929 15:52:48.523334 10681 fs_manager.cc:359] Metadata directory not provided
I20260929 15:52:48.523475 10681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:52:48.523761 10681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:52:48.526650 10681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "c128f177b1ca4c55b6ca071aed1f214e"
format_stamp: "Formatted at 2026-09-29 15:52:48 on dist-test-slave-r0ck"
server_key: "e3f518f76e9ef779a08162c5abacd415"
server_key_iv: "78c93118fd669fd85292e2b3f90dd321"
server_key_version: "encryptionkey@0"
I20260929 15:52:48.527395 10681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "c128f177b1ca4c55b6ca071aed1f214e"
format_stamp: "Formatted at 2026-09-29 15:52:48 on dist-test-slave-r0ck"
server_key: "e3f518f76e9ef779a08162c5abacd415"
server_key_iv: "78c93118fd669fd85292e2b3f90dd321"
server_key_version: "encryptionkey@0"
I20260929 15:52:48.532284 10681 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 15:52:48.535480 10696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:48.536706 10681 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 15:52:48.536895 10681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "c128f177b1ca4c55b6ca071aed1f214e"
format_stamp: "Formatted at 2026-09-29 15:52:48 on dist-test-slave-r0ck"
server_key: "e3f518f76e9ef779a08162c5abacd415"
server_key_iv: "78c93118fd669fd85292e2b3f90dd321"
server_key_version: "encryptionkey@0"
I20260929 15:52:48.537076 10681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:52:48.553014 10681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:52:48.556836 10681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:52:48.557103 10681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:52:48.574862 10681 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41961
I20260929 15:52:48.574900 10747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41961 every 8 connection(s)
I20260929 15:52:48.576184 10681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:52:48.579689 10748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:52:48.582469 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 10681
I20260929 15:52:48.582696 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:52:48.583485 388 external_mini_cluster.cc:1428] Setting key c9df32dd44b4dd538aab48ef8186fe3f
I20260929 15:52:48.591842 10748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c128f177b1ca4c55b6ca071aed1f214e: Bootstrap starting.
I20260929 15:52:48.595476 10748 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c128f177b1ca4c55b6ca071aed1f214e: Neither blocks nor log segments found. Creating new log.
I20260929 15:52:48.596746 10748 log.cc:824] T 00000000000000000000000000000000 P c128f177b1ca4c55b6ca071aed1f214e: Log is configured to *not* fsync() on all Append() calls
Sep 29 15:52:48 dist-test-slave-r0ck krb5kdc[10668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:52:48.599833 10748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c128f177b1ca4c55b6ca071aed1f214e: No bootstrap required, opened a new log
I20260929 15:52:48.607239 10751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:48.585518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36770 (local address 127.0.97.62:41961)
0929 15:52:48.586259 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:48.586270 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:48.586309 (+ 39us) server_negotiation.cc:407] Connection header received
0929 15:52:48.587917 (+ 1608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:48.587936 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:48.588289 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:48.588599 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:48.590333 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:48.591583 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:48.593329 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:48.594026 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:48.594148 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:48.598836 (+ 4688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:48.598845 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:48.598860 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:48.598907 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:48.602828 (+ 3921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:48.603480 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:48.603488 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:48.603497 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:48.603692 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:48.604063 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:48.604068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:48.604072 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:48.604682 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:48.604995 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:48.605410 (+ 415us) server_negotiation.cc:299] Negotiation successful
0929 15:52:48.605984 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":352,"threads_started":1}
I20260929 15:52:48.610090 10748 raft_consensus.cc:348] T 00000000000000000000000000000000 P c128f177b1ca4c55b6ca071aed1f214e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c128f177b1ca4c55b6ca071aed1f214e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41961 } }
I20260929 15:52:48.610414 10748 raft_consensus.cc:374] T 00000000000000000000000000000000 P c128f177b1ca4c55b6ca071aed1f214e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:52:48.610525 10748 raft_consensus.cc:729] T 00000000000000000000000000000000 P c128f177b1ca4c55b6ca071aed1f214e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c128f177b1ca4c55b6ca071aed1f214e, State: Initialized, Role: FOLLOWER
I20260929 15:52:48.611035 10748 consensus_queue.cc:260] T 00000000000000000000000000000000 P c128f177b1ca4c55b6ca071aed1f214e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c128f177b1ca4c55b6ca071aed1f214e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41961 } }
I20260929 15:52:48.611250 10748 raft_consensus.cc:388] T 00000000000000000000000000000000 P c128f177b1ca4c55b6ca071aed1f214e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:52:48.611383 10748 raft_consensus.cc:482] T 00000000000000000000000000000000 P c128f177b1ca4c55b6ca071aed1f214e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:52:48.611557 10748 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c128f177b1ca4c55b6ca071aed1f214e [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:52:48.613688 10748 raft_consensus.cc:504] T 00000000000000000000000000000000 P c128f177b1ca4c55b6ca071aed1f214e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c128f177b1ca4c55b6ca071aed1f214e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41961 } }
I20260929 15:52:48.614148 10748 leader_election.cc:302] T 00000000000000000000000000000000 P c128f177b1ca4c55b6ca071aed1f214e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c128f177b1ca4c55b6ca071aed1f214e; no voters:
I20260929 15:52:48.614773 10748 leader_election.cc:288] T 00000000000000000000000000000000 P c128f177b1ca4c55b6ca071aed1f214e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:52:48.614913 10753 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c128f177b1ca4c55b6ca071aed1f214e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:52:48.616200 10753 raft_consensus.cc:686] T 00000000000000000000000000000000 P c128f177b1ca4c55b6ca071aed1f214e [term 1 LEADER]: Becoming Leader. State: Replica: c128f177b1ca4c55b6ca071aed1f214e, State: Running, Role: LEADER
I20260929 15:52:48.616681 10753 consensus_queue.cc:237] T 00000000000000000000000000000000 P c128f177b1ca4c55b6ca071aed1f214e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c128f177b1ca4c55b6ca071aed1f214e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41961 } }
I20260929 15:52:48.616984 10748 sys_catalog.cc:564] T 00000000000000000000000000000000 P c128f177b1ca4c55b6ca071aed1f214e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:52:48.620199 10754 sys_catalog.cc:455] T 00000000000000000000000000000000 P c128f177b1ca4c55b6ca071aed1f214e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c128f177b1ca4c55b6ca071aed1f214e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c128f177b1ca4c55b6ca071aed1f214e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41961 } } }
I20260929 15:52:48.620605 10754 sys_catalog.cc:458] T 00000000000000000000000000000000 P c128f177b1ca4c55b6ca071aed1f214e [sys.catalog]: This master's current role is: LEADER
I20260929 15:52:48.620139 10755 sys_catalog.cc:455] T 00000000000000000000000000000000 P c128f177b1ca4c55b6ca071aed1f214e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c128f177b1ca4c55b6ca071aed1f214e. Latest consensus state: current_term: 1 leader_uuid: "c128f177b1ca4c55b6ca071aed1f214e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c128f177b1ca4c55b6ca071aed1f214e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41961 } } }
I20260929 15:52:48.621326 10755 sys_catalog.cc:458] T 00000000000000000000000000000000 P c128f177b1ca4c55b6ca071aed1f214e [sys.catalog]: This master's current role is: LEADER
I20260929 15:52:48.628057 10759 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:52:48.633399 10759 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:52:48.642330 10759 catalog_manager.cc:1347] Generated new cluster ID: b158f11bc9984c8b977f46f1d86dd086
I20260929 15:52:48.642437 10759 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:52:48.659090 10759 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:52:48.660794 10759 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:52:48.673867 10759 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c128f177b1ca4c55b6ca071aed1f214e: Generated new TSK 0
I20260929 15:52:48.674820 10759 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 15:52:48.734948 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41961
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:52:49.058564 10774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:52:49.058904 10774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:52:49.059063 10774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:52:49.066275 10774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:52:49.066424 10774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:52:49.066605 10774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:52:49.077500 10774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:52:49.078857 10774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:52:49.080159 10774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:52:49.095037 10779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:49.095084 10780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:49.095126 10782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:52:49.096167 10774 server_base.cc:1034] running on GCE node
I20260929 15:52:49.096786 10774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:52:49.097810 10774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:52:49.099125 10774 hybrid_clock.cc:648] HybridClock initialized: now 1790697169099060 us; error 49 us; skew 500 ppm
I20260929 15:52:49.099470 10774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:52:49 dist-test-slave-r0ck krb5kdc[10668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:52:49.102800 10774 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:52:49.104490 10774 webserver.cc:460] Webserver started at http://127.0.97.1:42895/ using document root <none> and password file <none>
I20260929 15:52:49.105192 10774 fs_manager.cc:359] Metadata directory not provided
I20260929 15:52:49.105389 10774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:52:49.105718 10774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:52:49.108561 10774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-0/data/instance:
uuid: "651c8f05d8ce4e40a04c26c2b4296c70"
format_stamp: "Formatted at 2026-09-29 15:52:49 on dist-test-slave-r0ck"
server_key: "e05f9ddd5a98d0f2ae03445e0ccbbb52"
server_key_iv: "96d6bb1a12c95958358124cc5f390669"
server_key_version: "encryptionkey@0"
I20260929 15:52:49.109326 10774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "651c8f05d8ce4e40a04c26c2b4296c70"
format_stamp: "Formatted at 2026-09-29 15:52:49 on dist-test-slave-r0ck"
server_key: "e05f9ddd5a98d0f2ae03445e0ccbbb52"
server_key_iv: "96d6bb1a12c95958358124cc5f390669"
server_key_version: "encryptionkey@0"
I20260929 15:52:49.114464 10774 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260929 15:52:49.118068 10789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:49.119381 10774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:52:49.119572 10774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-0/wal
uuid: "651c8f05d8ce4e40a04c26c2b4296c70"
format_stamp: "Formatted at 2026-09-29 15:52:49 on dist-test-slave-r0ck"
server_key: "e05f9ddd5a98d0f2ae03445e0ccbbb52"
server_key_iv: "96d6bb1a12c95958358124cc5f390669"
server_key_version: "encryptionkey@0"
I20260929 15:52:49.119774 10774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:52:49.151007 10774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:52:49.155170 10774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:52:49.155462 10774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:52:49.156385 10774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:52:49.157729 10774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:52:49.157828 10774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:49.157982 10774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:52:49.158061 10774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:49.191174 10774 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44321
I20260929 15:52:49.191236 10901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44321 every 8 connection(s)
I20260929 15:52:49.192420 10774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:52:49.199632 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 10774
I20260929 15:52:49.199872 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:52:49.200469 388 external_mini_cluster.cc:1428] Setting key ca75b7f770b2fad884296e7426e19178
Sep 29 15:52:49 dist-test-slave-r0ck krb5kdc[10668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:52:49.212522 10904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:49.194862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39483 (local address 127.0.97.62:41961)
0929 15:52:49.195296 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:49.195300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:49.196133 (+ 833us) server_negotiation.cc:407] Connection header received
0929 15:52:49.197951 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:49.197956 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:49.198045 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:49.198165 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:49.199797 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:49.200614 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:49.202333 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:49.202977 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:49.203108 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:49.206527 (+ 3419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:49.206532 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:49.206535 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:49.206564 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:49.209124 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:49.209983 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:49.209990 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:49.209996 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:49.210150 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:49.210667 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:49.210672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:49.210674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:49.211016 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:49.211160 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:49.211808 (+ 648us) server_negotiation.cc:299] Negotiation successful
0929 15:52:49.212057 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":233,"threads_started":1}
I20260929 15:52:49.213085 10905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:49.195325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41961 (local address 127.0.97.1:39483)
0929 15:52:49.195985 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:49.196021 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:49.197686 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:49.198383 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:49.198391 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:49.198865 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:49.199570 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:49.199597 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:49.200776 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:49.200782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:49.202181 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:49.202192 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:49.203192 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:49.203199 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:49.203427 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:49.204116 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:49.204137 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:49.206310 (+ 2173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:49.209268 (+ 2958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:49.209277 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:49.209293 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:49.209824 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:49.210281 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:49.210287 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:49.210292 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:49.210539 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:49.211139 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:49.211145 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:49.211438 (+ 293us) client_negotiation.cc:769] Sending connection context
0929 15:52:49.211652 (+ 214us) client_negotiation.cc:240] Negotiation successful
0929 15:52:49.212010 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":272,"threads_started":1}
I20260929 15:52:49.214654 10902 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41961
I20260929 15:52:49.214960 10902 heartbeater.cc:464] Registering TS with master...
I20260929 15:52:49.215626 10902 heartbeater.cc:511] Master 127.0.97.62:41961 requested a full tablet report, sending...
I20260929 15:52:49.217813 10713 ts_manager.cc:194] Registered new tserver with Master: 651c8f05d8ce4e40a04c26c2b4296c70 (127.0.97.1:44321)
I20260929 15:52:49.219286 10713 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:39483
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 15:52:49.256492 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41961
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:52:49.600431 10908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:52:49.600690 10908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:52:49.600850 10908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:52:49.609783 10908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:52:49.609930 10908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:52:49.610179 10908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:52:49.620728 10908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:52:49.622102 10908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:52:49.623418 10908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:52:49.635974 10913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:49.636314 10916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:49.635946 10914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:52:49.636814 10908 server_base.cc:1034] running on GCE node
I20260929 15:52:49.637457 10908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:52:49.638356 10908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:52:49.639581 10908 hybrid_clock.cc:648] HybridClock initialized: now 1790697169639541 us; error 51 us; skew 500 ppm
I20260929 15:52:49.639889 10908 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:52:49 dist-test-slave-r0ck krb5kdc[10668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:52:49.643023 10908 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:52:49.644440 10908 webserver.cc:460] Webserver started at http://127.0.97.2:44061/ using document root <none> and password file <none>
I20260929 15:52:49.645119 10908 fs_manager.cc:359] Metadata directory not provided
I20260929 15:52:49.645272 10908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:52:49.645609 10908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:52:49.648507 10908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-1/data/instance:
uuid: "20ad4e94f216486d95f9aaad1e3261d2"
format_stamp: "Formatted at 2026-09-29 15:52:49 on dist-test-slave-r0ck"
server_key: "8b7cc0ba0bc54cf014f78c9b257df86c"
server_key_iv: "0606e697e61625c4cceb35c031b2264f"
server_key_version: "encryptionkey@0"
I20260929 15:52:49.649299 10908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "20ad4e94f216486d95f9aaad1e3261d2"
format_stamp: "Formatted at 2026-09-29 15:52:49 on dist-test-slave-r0ck"
server_key: "8b7cc0ba0bc54cf014f78c9b257df86c"
server_key_iv: "0606e697e61625c4cceb35c031b2264f"
server_key_version: "encryptionkey@0"
I20260929 15:52:49.655360 10908 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260929 15:52:49.659523 10923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:49.661031 10908 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 15:52:49.661228 10908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-1/wal
uuid: "20ad4e94f216486d95f9aaad1e3261d2"
format_stamp: "Formatted at 2026-09-29 15:52:49 on dist-test-slave-r0ck"
server_key: "8b7cc0ba0bc54cf014f78c9b257df86c"
server_key_iv: "0606e697e61625c4cceb35c031b2264f"
server_key_version: "encryptionkey@0"
I20260929 15:52:49.661494 10908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:52:49.677235 10908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:52:49.681457 10908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:52:49.681733 10908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:52:49.682682 10908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:52:49.684011 10908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:52:49.684114 10908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:49.684254 10908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:52:49.684335 10908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:49.718005 10908 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46171
I20260929 15:52:49.718060 11035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46171 every 8 connection(s)
I20260929 15:52:49.719358 10908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:52:49.724617 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 10908
I20260929 15:52:49.724815 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:52:49.725365 388 external_mini_cluster.cc:1428] Setting key a156ea9021ef66da3edda6b10f57d246
Sep 29 15:52:49 dist-test-slave-r0ck krb5kdc[10668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:52:49.740821 11038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:49.721848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60527 (local address 127.0.97.62:41961)
0929 15:52:49.722278 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:49.722282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:49.723227 (+ 945us) server_negotiation.cc:407] Connection header received
0929 15:52:49.725644 (+ 2417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:49.725649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:49.725717 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:49.725818 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:49.727284 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:49.728048 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:49.730578 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:49.730994 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:49.731113 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:49.735201 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:49.735206 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:49.735210 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:49.735239 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:49.737776 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:49.738494 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:49.738499 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:49.738501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:49.738626 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:49.739075 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:49.739081 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:49.739084 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:49.739335 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:49.739493 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:49.740208 (+ 715us) server_negotiation.cc:299] Negotiation successful
0929 15:52:49.740423 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":230,"threads_started":1}
I20260929 15:52:49.741832 11039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:49.722231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41961 (local address 127.0.97.2:60527)
0929 15:52:49.723067 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:49.723104 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:49.725333 (+ 2229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:49.726073 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:49.726081 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:49.726561 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:49.727089 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:49.727105 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:49.728214 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:49.728222 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:49.730432 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:49.730449 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:49.731218 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:49.731228 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:49.731547 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:49.732295 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:49.732327 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:49.734957 (+ 2630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:49.737922 (+ 2965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:49.737937 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:49.737949 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:49.738356 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:49.738758 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:49.738763 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:49.738773 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:49.738953 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:49.739468 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:49.739476 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:49.739830 (+ 354us) client_negotiation.cc:769] Sending connection context
0929 15:52:49.740103 (+ 273us) client_negotiation.cc:240] Negotiation successful
0929 15:52:49.740615 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":289,"threads_started":1}
I20260929 15:52:49.743263 11036 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41961
I20260929 15:52:49.743594 11036 heartbeater.cc:464] Registering TS with master...
I20260929 15:52:49.744470 11036 heartbeater.cc:511] Master 127.0.97.62:41961 requested a full tablet report, sending...
I20260929 15:52:49.746275 10713 ts_manager.cc:194] Registered new tserver with Master: 20ad4e94f216486d95f9aaad1e3261d2 (127.0.97.2:46171)
I20260929 15:52:49.747557 10713 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:60527
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 15:52:49.781195 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41961
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:52:50.107640 11042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:52:50.107888 11042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:52:50.107978 11042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:52:50.114962 11042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:52:50.115065 11042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:52:50.115175 11042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:52:50.124699 11042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:52:50.125967 11042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:52:50.127132 11042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:52:50.139389 11048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:50.139492 11047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:52:50.139467 11050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:52:50.140303 11042 server_base.cc:1034] running on GCE node
I20260929 15:52:50.140848 11042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:52:50.141813 11042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:52:50.143064 11042 hybrid_clock.cc:648] HybridClock initialized: now 1790697170143021 us; error 63 us; skew 500 ppm
I20260929 15:52:50.143397 11042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:52:50 dist-test-slave-r0ck krb5kdc[10668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:52:50.146585 11042 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:52:50.148020 11042 webserver.cc:460] Webserver started at http://127.0.97.3:40851/ using document root <none> and password file <none>
I20260929 15:52:50.148722 11042 fs_manager.cc:359] Metadata directory not provided
I20260929 15:52:50.148866 11042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:52:50.149166 11042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:52:50.152064 11042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-2/data/instance:
uuid: "2891c04c2dba42a29b438e9237e52537"
format_stamp: "Formatted at 2026-09-29 15:52:50 on dist-test-slave-r0ck"
server_key: "99bf8e4b6288a76334f4a0f4257b1e73"
server_key_iv: "d67e12e568b4fafbec425e165252d2a6"
server_key_version: "encryptionkey@0"
I20260929 15:52:50.152838 11042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "2891c04c2dba42a29b438e9237e52537"
format_stamp: "Formatted at 2026-09-29 15:52:50 on dist-test-slave-r0ck"
server_key: "99bf8e4b6288a76334f4a0f4257b1e73"
server_key_iv: "d67e12e568b4fafbec425e165252d2a6"
server_key_version: "encryptionkey@0"
I20260929 15:52:50.158015 11042 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 15:52:50.161770 11057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:50.163216 11042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 15:52:50.163357 11042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-2/wal
uuid: "2891c04c2dba42a29b438e9237e52537"
format_stamp: "Formatted at 2026-09-29 15:52:50 on dist-test-slave-r0ck"
server_key: "99bf8e4b6288a76334f4a0f4257b1e73"
server_key_iv: "d67e12e568b4fafbec425e165252d2a6"
server_key_version: "encryptionkey@0"
I20260929 15:52:50.163522 11042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:52:50.179842 11042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:52:50.183897 11042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:52:50.184127 11042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:52:50.184949 11042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:52:50.186228 11042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:52:50.186298 11042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:50.186379 11042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:52:50.186422 11042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:52:50.218940 11042 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43123
I20260929 15:52:50.218972 11169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43123 every 8 connection(s)
I20260929 15:52:50.220168 11042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:52:50.221930 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 11042
I20260929 15:52:50.222124 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:52:50.222301 10902 heartbeater.cc:503] Master 127.0.97.62:41961 was elected leader, sending a full tablet report...
I20260929 15:52:50.222687 388 external_mini_cluster.cc:1428] Setting key b395a46148a28d491ede8ade0f513459
Sep 29 15:52:50 dist-test-slave-r0ck krb5kdc[10668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:52:50.240614 11038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:50.223452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52153 (local address 127.0.97.62:41961)
0929 15:52:50.223627 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:50.223633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:50.224644 (+ 1011us) server_negotiation.cc:407] Connection header received
0929 15:52:50.226631 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:50.226635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:50.226703 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:50.226783 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:50.228247 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:50.229034 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:50.230635 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:50.231063 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:50.231161 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:50.234666 (+ 3505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:50.234671 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:50.234675 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:50.234711 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:50.237774 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:50.238542 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:50.238546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:50.238549 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:50.238661 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:50.239026 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:50.239030 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:50.239032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:50.239292 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:50.239397 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:50.240176 (+ 779us) server_negotiation.cc:299] Negotiation successful
0929 15:52:50.240367 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 15:52:50.241633 11172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:50.223528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41961 (local address 127.0.97.3:52153)
0929 15:52:50.224494 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:50.224548 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:50.226409 (+ 1861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:50.226937 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:50.226945 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:50.227409 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:52:50.228007 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:50.228024 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:50.229186 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:50.229191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:50.230494 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:50.230506 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:50.231296 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:50.231306 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:50.231615 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:50.232238 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:52:50.232259 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:52:50.234470 (+ 2211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:52:50.237946 (+ 3476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:50.237954 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:50.237970 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:50.238397 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:50.238754 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:52:50.238757 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:52:50.238759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:52:50.238930 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:52:50.239402 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:52:50.239407 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:52:50.239721 (+ 314us) client_negotiation.cc:769] Sending connection context
0929 15:52:50.240006 (+ 285us) client_negotiation.cc:240] Negotiation successful
0929 15:52:50.240446 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":330,"threads_started":1}
I20260929 15:52:50.243000 11170 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41961
I20260929 15:52:50.243247 11170 heartbeater.cc:464] Registering TS with master...
I20260929 15:52:50.243898 11170 heartbeater.cc:511] Master 127.0.97.62:41961 requested a full tablet report, sending...
I20260929 15:52:50.245215 10712 ts_manager.cc:194] Registered new tserver with Master: 2891c04c2dba42a29b438e9237e52537 (127.0.97.3:43123)
I20260929 15:52:50.246079 10712 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52153
I20260929 15:52:50.250121 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:52:50.267079 11038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:50.254670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36786 (local address 127.0.97.62:41961)
0929 15:52:50.254871 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:50.254875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:50.255280 (+ 405us) server_negotiation.cc:407] Connection header received
0929 15:52:50.255414 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:50.255418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:50.255485 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:50.255570 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:50.256915 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:50.257766 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:50.259015 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:50.259431 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:50.259569 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:50.261878 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:50.261884 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:50.261889 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:50.261929 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:50.264397 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:50.265074 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:50.265081 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:50.265084 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:50.265223 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:50.265622 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:50.265632 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:50.265634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:50.265928 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:50.266222 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:50.266520 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 15:52:50.266725 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 15:52:50.271612 10712 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36786:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:52:50.278264 10712 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:52:50.314579 11184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:50.289918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42932 (local address 127.0.97.3:43123)
0929 15:52:50.305039 (+ 15121us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:50.305051 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:50.305093 (+ 42us) server_negotiation.cc:407] Connection header received
0929 15:52:50.305323 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:50.305331 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:50.305617 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:50.306614 (+ 997us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:52:50.306889 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:50.308501 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:50.311629 (+ 3128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:50.313375 (+ 1746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:50.313504 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:50.313747 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:50.314083 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 15:52:50.314249 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14922,"thread_start_us":14735,"threads_started":1}
I20260929 15:52:50.314626 11182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:50.289679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43123 (local address 127.0.0.1:42932)
0929 15:52:50.304771 (+ 15092us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:50.304822 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:50.305028 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:50.305864 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:50.305871 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:50.305915 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:52:50.306418 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:50.306432 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:50.308682 (+ 2250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:50.308687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:50.311394 (+ 2707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:50.311410 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:50.313565 (+ 2155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:50.313572 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:50.313839 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:50.313889 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 15:52:50.314033 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 15:52:50.314220 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14780,"mutex_wait_us":16,"thread_start_us":15201,"threads_started":1}
I20260929 15:52:50.317034 11187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:50.292742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59770 (local address 127.0.97.1:44321)
0929 15:52:50.305967 (+ 13225us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:50.305977 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:50.308116 (+ 2139us) server_negotiation.cc:407] Connection header received
0929 15:52:50.308244 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:50.308253 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:50.308696 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:50.309025 (+ 329us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:52:50.310205 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:50.312055 (+ 1850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:50.314391 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:50.315851 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:50.315979 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:50.316198 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:50.316535 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 15:52:50.316694 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13056,"thread_start_us":12897,"threads_started":1}
I20260929 15:52:50.317060 11186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:50.289817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46171 (local address 127.0.0.1:50032)
0929 15:52:50.306418 (+ 16601us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:50.306442 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:50.306557 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:50.308199 (+ 1642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:50.308204 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:50.308229 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:52:50.308646 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:50.308674 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:50.310711 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:50.310718 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:50.314008 (+ 3290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:50.314026 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:50.316080 (+ 2054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:50.316087 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:50.316307 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:50.316401 (+ 94us) client_negotiation.cc:769] Sending connection context
0929 15:52:50.316490 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 15:52:50.316624 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16435,"mutex_wait_us":65,"thread_start_us":17435,"threads_started":1}
I20260929 15:52:50.317076 11183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:50.291991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44321 (local address 127.0.0.1:59770)
0929 15:52:50.307684 (+ 15693us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:50.307712 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:50.307918 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:50.309099 (+ 1181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:50.309104 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:50.309132 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:52:50.309804 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:50.309816 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:50.312283 (+ 2467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:50.312288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:50.314209 (+ 1921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:50.314220 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:50.316122 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:50.316127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:50.316353 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:50.316385 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 15:52:50.316479 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 15:52:50.316601 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15482,"mutex_wait_us":29,"thread_start_us":14124,"threads_started":1}
I20260929 15:52:50.317731 11185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:50.290293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50032 (local address 127.0.97.2:46171)
0929 15:52:50.307039 (+ 16746us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:50.307047 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:50.307071 (+ 24us) server_negotiation.cc:407] Connection header received
0929 15:52:50.307169 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:50.307176 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:50.307793 (+ 617us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:50.307988 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:52:50.308874 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:50.310478 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:50.314191 (+ 3713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:50.315630 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:50.315745 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:50.315916 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:50.317198 (+ 1282us) server_negotiation.cc:299] Negotiation successful
0929 15:52:50.317383 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16592,"thread_start_us":16457,"threads_started":1}
I20260929 15:52:50.319087 11105 tablet_service.cc:1467] Processing CreateTablet for tablet 42ce0f50dfbd4235b15ba0c811d50c6c (DEFAULT_TABLE table=test-table [id=da868b3576ff49ce98cee7bf6bd550a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:52:50.320506 11105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42ce0f50dfbd4235b15ba0c811d50c6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:52:50.320767 10971 tablet_service.cc:1467] Processing CreateTablet for tablet 42ce0f50dfbd4235b15ba0c811d50c6c (DEFAULT_TABLE table=test-table [id=da868b3576ff49ce98cee7bf6bd550a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:52:50.320767 10837 tablet_service.cc:1467] Processing CreateTablet for tablet 42ce0f50dfbd4235b15ba0c811d50c6c (DEFAULT_TABLE table=test-table [id=da868b3576ff49ce98cee7bf6bd550a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:52:50.322024 10971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42ce0f50dfbd4235b15ba0c811d50c6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:52:50.322023 10837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42ce0f50dfbd4235b15ba0c811d50c6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:52:50.332075 11188 tablet_bootstrap.cc:492] T 42ce0f50dfbd4235b15ba0c811d50c6c P 2891c04c2dba42a29b438e9237e52537: Bootstrap starting.
I20260929 15:52:50.335443 11188 tablet_bootstrap.cc:654] T 42ce0f50dfbd4235b15ba0c811d50c6c P 2891c04c2dba42a29b438e9237e52537: Neither blocks nor log segments found. Creating new log.
I20260929 15:52:50.336299 11190 tablet_bootstrap.cc:492] T 42ce0f50dfbd4235b15ba0c811d50c6c P 20ad4e94f216486d95f9aaad1e3261d2: Bootstrap starting.
I20260929 15:52:50.337040 11189 tablet_bootstrap.cc:492] T 42ce0f50dfbd4235b15ba0c811d50c6c P 651c8f05d8ce4e40a04c26c2b4296c70: Bootstrap starting.
I20260929 15:52:50.337172 11188 log.cc:824] T 42ce0f50dfbd4235b15ba0c811d50c6c P 2891c04c2dba42a29b438e9237e52537: Log is configured to *not* fsync() on all Append() calls
I20260929 15:52:50.341197 11189 tablet_bootstrap.cc:654] T 42ce0f50dfbd4235b15ba0c811d50c6c P 651c8f05d8ce4e40a04c26c2b4296c70: Neither blocks nor log segments found. Creating new log.
I20260929 15:52:50.342499 11189 log.cc:824] T 42ce0f50dfbd4235b15ba0c811d50c6c P 651c8f05d8ce4e40a04c26c2b4296c70: Log is configured to *not* fsync() on all Append() calls
I20260929 15:52:50.342855 11190 tablet_bootstrap.cc:654] T 42ce0f50dfbd4235b15ba0c811d50c6c P 20ad4e94f216486d95f9aaad1e3261d2: Neither blocks nor log segments found. Creating new log.
I20260929 15:52:50.342875 11188 tablet_bootstrap.cc:492] T 42ce0f50dfbd4235b15ba0c811d50c6c P 2891c04c2dba42a29b438e9237e52537: No bootstrap required, opened a new log
I20260929 15:52:50.343171 11188 ts_tablet_manager.cc:1397] T 42ce0f50dfbd4235b15ba0c811d50c6c P 2891c04c2dba42a29b438e9237e52537: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.002s
I20260929 15:52:50.344028 11190 log.cc:824] T 42ce0f50dfbd4235b15ba0c811d50c6c P 20ad4e94f216486d95f9aaad1e3261d2: Log is configured to *not* fsync() on all Append() calls
I20260929 15:52:50.345436 11189 tablet_bootstrap.cc:492] T 42ce0f50dfbd4235b15ba0c811d50c6c P 651c8f05d8ce4e40a04c26c2b4296c70: No bootstrap required, opened a new log
I20260929 15:52:50.345677 11189 ts_tablet_manager.cc:1397] T 42ce0f50dfbd4235b15ba0c811d50c6c P 651c8f05d8ce4e40a04c26c2b4296c70: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260929 15:52:50.347482 11190 tablet_bootstrap.cc:492] T 42ce0f50dfbd4235b15ba0c811d50c6c P 20ad4e94f216486d95f9aaad1e3261d2: No bootstrap required, opened a new log
I20260929 15:52:50.347740 11190 ts_tablet_manager.cc:1397] T 42ce0f50dfbd4235b15ba0c811d50c6c P 20ad4e94f216486d95f9aaad1e3261d2: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.005s
I20260929 15:52:50.353797 11188 raft_consensus.cc:348] T 42ce0f50dfbd4235b15ba0c811d50c6c P 2891c04c2dba42a29b438e9237e52537 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2891c04c2dba42a29b438e9237e52537" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43123 } } peers { permanent_uuid: "20ad4e94f216486d95f9aaad1e3261d2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46171 } } peers { permanent_uuid: "651c8f05d8ce4e40a04c26c2b4296c70" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44321 } }
I20260929 15:52:50.354271 11188 raft_consensus.cc:374] T 42ce0f50dfbd4235b15ba0c811d50c6c P 2891c04c2dba42a29b438e9237e52537 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:52:50.354413 11188 raft_consensus.cc:729] T 42ce0f50dfbd4235b15ba0c811d50c6c P 2891c04c2dba42a29b438e9237e52537 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2891c04c2dba42a29b438e9237e52537, State: Initialized, Role: FOLLOWER
I20260929 15:52:50.355042 11190 raft_consensus.cc:348] T 42ce0f50dfbd4235b15ba0c811d50c6c P 20ad4e94f216486d95f9aaad1e3261d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2891c04c2dba42a29b438e9237e52537" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43123 } } peers { permanent_uuid: "20ad4e94f216486d95f9aaad1e3261d2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46171 } } peers { permanent_uuid: "651c8f05d8ce4e40a04c26c2b4296c70" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44321 } }
I20260929 15:52:50.355043 11188 consensus_queue.cc:260] T 42ce0f50dfbd4235b15ba0c811d50c6c P 2891c04c2dba42a29b438e9237e52537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2891c04c2dba42a29b438e9237e52537" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43123 } } peers { permanent_uuid: "20ad4e94f216486d95f9aaad1e3261d2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46171 } } peers { permanent_uuid: "651c8f05d8ce4e40a04c26c2b4296c70" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44321 } }
I20260929 15:52:50.355379 11190 raft_consensus.cc:374] T 42ce0f50dfbd4235b15ba0c811d50c6c P 20ad4e94f216486d95f9aaad1e3261d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:52:50.355499 11190 raft_consensus.cc:729] T 42ce0f50dfbd4235b15ba0c811d50c6c P 20ad4e94f216486d95f9aaad1e3261d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20ad4e94f216486d95f9aaad1e3261d2, State: Initialized, Role: FOLLOWER
I20260929 15:52:50.356204 11170 heartbeater.cc:503] Master 127.0.97.62:41961 was elected leader, sending a full tablet report...
I20260929 15:52:50.356247 11190 consensus_queue.cc:260] T 42ce0f50dfbd4235b15ba0c811d50c6c P 20ad4e94f216486d95f9aaad1e3261d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2891c04c2dba42a29b438e9237e52537" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43123 } } peers { permanent_uuid: "20ad4e94f216486d95f9aaad1e3261d2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46171 } } peers { permanent_uuid: "651c8f05d8ce4e40a04c26c2b4296c70" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44321 } }
I20260929 15:52:50.356208 11189 raft_consensus.cc:348] T 42ce0f50dfbd4235b15ba0c811d50c6c P 651c8f05d8ce4e40a04c26c2b4296c70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2891c04c2dba42a29b438e9237e52537" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43123 } } peers { permanent_uuid: "20ad4e94f216486d95f9aaad1e3261d2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46171 } } peers { permanent_uuid: "651c8f05d8ce4e40a04c26c2b4296c70" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44321 } }
I20260929 15:52:50.356607 11189 raft_consensus.cc:374] T 42ce0f50dfbd4235b15ba0c811d50c6c P 651c8f05d8ce4e40a04c26c2b4296c70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:52:50.356775 11189 raft_consensus.cc:729] T 42ce0f50dfbd4235b15ba0c811d50c6c P 651c8f05d8ce4e40a04c26c2b4296c70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 651c8f05d8ce4e40a04c26c2b4296c70, State: Initialized, Role: FOLLOWER
I20260929 15:52:50.357138 11188 ts_tablet_manager.cc:1428] T 42ce0f50dfbd4235b15ba0c811d50c6c P 2891c04c2dba42a29b438e9237e52537: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260929 15:52:50.357542 11189 consensus_queue.cc:260] T 42ce0f50dfbd4235b15ba0c811d50c6c P 651c8f05d8ce4e40a04c26c2b4296c70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2891c04c2dba42a29b438e9237e52537" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43123 } } peers { permanent_uuid: "20ad4e94f216486d95f9aaad1e3261d2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46171 } } peers { permanent_uuid: "651c8f05d8ce4e40a04c26c2b4296c70" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44321 } }
I20260929 15:52:50.358091 11036 heartbeater.cc:503] Master 127.0.97.62:41961 was elected leader, sending a full tablet report...
I20260929 15:52:50.360682 11190 ts_tablet_manager.cc:1428] T 42ce0f50dfbd4235b15ba0c811d50c6c P 20ad4e94f216486d95f9aaad1e3261d2: Time spent starting tablet: real 0.013s user 0.010s sys 0.002s
I20260929 15:52:50.365669 11189 ts_tablet_manager.cc:1428] T 42ce0f50dfbd4235b15ba0c811d50c6c P 651c8f05d8ce4e40a04c26c2b4296c70: Time spent starting tablet: real 0.020s user 0.014s sys 0.005s
W20260929 15:52:50.444778 10903 tablet.cc:2369] T 42ce0f50dfbd4235b15ba0c811d50c6c P 651c8f05d8ce4e40a04c26c2b4296c70: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:52:50.471299 11037 tablet.cc:2369] T 42ce0f50dfbd4235b15ba0c811d50c6c P 20ad4e94f216486d95f9aaad1e3261d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:52:50.471737 11171 tablet.cc:2369] T 42ce0f50dfbd4235b15ba0c811d50c6c P 2891c04c2dba42a29b438e9237e52537: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:52:50.548360 11196 raft_consensus.cc:482] T 42ce0f50dfbd4235b15ba0c811d50c6c P 651c8f05d8ce4e40a04c26c2b4296c70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:52:50.548753 11196 raft_consensus.cc:504] T 42ce0f50dfbd4235b15ba0c811d50c6c P 651c8f05d8ce4e40a04c26c2b4296c70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2891c04c2dba42a29b438e9237e52537" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43123 } } peers { permanent_uuid: "20ad4e94f216486d95f9aaad1e3261d2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46171 } } peers { permanent_uuid: "651c8f05d8ce4e40a04c26c2b4296c70" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44321 } }
I20260929 15:52:50.550464 11196 leader_election.cc:288] T 42ce0f50dfbd4235b15ba0c811d50c6c P 651c8f05d8ce4e40a04c26c2b4296c70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2891c04c2dba42a29b438e9237e52537 (127.0.97.3:43123), 20ad4e94f216486d95f9aaad1e3261d2 (127.0.97.2:46171)
I20260929 15:52:50.559965 11185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:50.551093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37833 (local address 127.0.97.2:46171)
0929 15:52:50.551292 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:50.551300 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:50.552134 (+ 834us) server_negotiation.cc:407] Connection header received
0929 15:52:50.552356 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:50.552362 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:50.552438 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:50.552547 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:52:50.553420 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:50.554514 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:50.557596 (+ 3082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:50.558851 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:50.558989 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:50.559085 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:50.559572 (+ 487us) server_negotiation.cc:299] Negotiation successful
0929 15:52:50.559700 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 15:52:50.559958 11184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:50.550890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40283 (local address 127.0.97.3:43123)
0929 15:52:50.551115 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:50.551123 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:50.552134 (+ 1011us) server_negotiation.cc:407] Connection header received
0929 15:52:50.552397 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:50.552402 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:50.552476 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:50.552566 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:52:50.553420 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:50.554514 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:50.557600 (+ 3086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:50.558851 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:50.558989 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:50.559088 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:50.559572 (+ 484us) server_negotiation.cc:299] Negotiation successful
0929 15:52:50.559700 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 15:52:50.560179 11198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:50.550807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43123 (local address 127.0.97.1:40283)
0929 15:52:50.551984 (+ 1177us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:50.552006 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:50.552195 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:50.552590 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:50.552595 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:50.552627 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:52:50.553251 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:50.553265 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:50.554676 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:50.554682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:50.557454 (+ 2772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:50.557462 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:50.559127 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:50.559135 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:50.559367 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:50.559433 (+ 66us) client_negotiation.cc:769] Sending connection context
0929 15:52:50.559533 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 15:52:50.559689 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":958,"mutex_wait_us":308,"spinlock_wait_cycles":22912,"thread_start_us":684,"threads_started":1}
I20260929 15:52:50.560200 11197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:50.550876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46171 (local address 127.0.97.1:37833)
0929 15:52:50.551987 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0929 15:52:50.552006 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:52:50.552170 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:52:50.552590 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:52:50.552595 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:52:50.552644 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:52:50.553213 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:50.553236 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:50.554678 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:50.554682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:50.557430 (+ 2748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:52:50.557449 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:52:50.559143 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:52:50.559147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:52:50.559362 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:50.559433 (+ 71us) client_negotiation.cc:769] Sending connection context
0929 15:52:50.559533 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 15:52:50.559689 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":945,"mutex_wait_us":181,"spinlock_wait_cycles":33024,"thread_start_us":587,"threads_started":1}
I20260929 15:52:50.561304 11125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ce0f50dfbd4235b15ba0c811d50c6c" candidate_uuid: "651c8f05d8ce4e40a04c26c2b4296c70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2891c04c2dba42a29b438e9237e52537" is_pre_election: true
I20260929 15:52:50.561329 10991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ce0f50dfbd4235b15ba0c811d50c6c" candidate_uuid: "651c8f05d8ce4e40a04c26c2b4296c70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20ad4e94f216486d95f9aaad1e3261d2" is_pre_election: true
I20260929 15:52:50.561838 11125 raft_consensus.cc:2454] T 42ce0f50dfbd4235b15ba0c811d50c6c P 2891c04c2dba42a29b438e9237e52537 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 651c8f05d8ce4e40a04c26c2b4296c70 in term 0.
I20260929 15:52:50.561848 10991 raft_consensus.cc:2454] T 42ce0f50dfbd4235b15ba0c811d50c6c P 20ad4e94f216486d95f9aaad1e3261d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 651c8f05d8ce4e40a04c26c2b4296c70 in term 0.
I20260929 15:52:50.562515 10790 leader_election.cc:302] T 42ce0f50dfbd4235b15ba0c811d50c6c P 651c8f05d8ce4e40a04c26c2b4296c70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2891c04c2dba42a29b438e9237e52537, 651c8f05d8ce4e40a04c26c2b4296c70; no voters:
I20260929 15:52:50.562883 11196 raft_consensus.cc:2781] T 42ce0f50dfbd4235b15ba0c811d50c6c P 651c8f05d8ce4e40a04c26c2b4296c70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:52:50.563028 11196 raft_consensus.cc:482] T 42ce0f50dfbd4235b15ba0c811d50c6c P 651c8f05d8ce4e40a04c26c2b4296c70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:52:50.563148 11196 raft_consensus.cc:3037] T 42ce0f50dfbd4235b15ba0c811d50c6c P 651c8f05d8ce4e40a04c26c2b4296c70 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:52:50.565452 11196 raft_consensus.cc:504] T 42ce0f50dfbd4235b15ba0c811d50c6c P 651c8f05d8ce4e40a04c26c2b4296c70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2891c04c2dba42a29b438e9237e52537" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43123 } } peers { permanent_uuid: "20ad4e94f216486d95f9aaad1e3261d2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46171 } } peers { permanent_uuid: "651c8f05d8ce4e40a04c26c2b4296c70" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44321 } }
I20260929 15:52:50.566082 11196 leader_election.cc:288] T 42ce0f50dfbd4235b15ba0c811d50c6c P 651c8f05d8ce4e40a04c26c2b4296c70 [CANDIDATE]: Term 1 election: Requested vote from peers 2891c04c2dba42a29b438e9237e52537 (127.0.97.3:43123), 20ad4e94f216486d95f9aaad1e3261d2 (127.0.97.2:46171)
I20260929 15:52:50.566535 10991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ce0f50dfbd4235b15ba0c811d50c6c" candidate_uuid: "651c8f05d8ce4e40a04c26c2b4296c70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20ad4e94f216486d95f9aaad1e3261d2"
I20260929 15:52:50.566722 10991 raft_consensus.cc:3037] T 42ce0f50dfbd4235b15ba0c811d50c6c P 20ad4e94f216486d95f9aaad1e3261d2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:52:50.566817 11125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ce0f50dfbd4235b15ba0c811d50c6c" candidate_uuid: "651c8f05d8ce4e40a04c26c2b4296c70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2891c04c2dba42a29b438e9237e52537"
I20260929 15:52:50.567137 11125 raft_consensus.cc:3037] T 42ce0f50dfbd4235b15ba0c811d50c6c P 2891c04c2dba42a29b438e9237e52537 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:52:50.569451 10991 raft_consensus.cc:2454] T 42ce0f50dfbd4235b15ba0c811d50c6c P 20ad4e94f216486d95f9aaad1e3261d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 651c8f05d8ce4e40a04c26c2b4296c70 in term 1.
I20260929 15:52:50.569857 10792 leader_election.cc:302] T 42ce0f50dfbd4235b15ba0c811d50c6c P 651c8f05d8ce4e40a04c26c2b4296c70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20ad4e94f216486d95f9aaad1e3261d2, 651c8f05d8ce4e40a04c26c2b4296c70; no voters:
I20260929 15:52:50.570137 11196 raft_consensus.cc:2781] T 42ce0f50dfbd4235b15ba0c811d50c6c P 651c8f05d8ce4e40a04c26c2b4296c70 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:52:50.570436 11125 raft_consensus.cc:2454] T 42ce0f50dfbd4235b15ba0c811d50c6c P 2891c04c2dba42a29b438e9237e52537 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 651c8f05d8ce4e40a04c26c2b4296c70 in term 1.
I20260929 15:52:50.570631 11196 raft_consensus.cc:686] T 42ce0f50dfbd4235b15ba0c811d50c6c P 651c8f05d8ce4e40a04c26c2b4296c70 [term 1 LEADER]: Becoming Leader. State: Replica: 651c8f05d8ce4e40a04c26c2b4296c70, State: Running, Role: LEADER
I20260929 15:52:50.571353 11196 consensus_queue.cc:237] T 42ce0f50dfbd4235b15ba0c811d50c6c P 651c8f05d8ce4e40a04c26c2b4296c70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2891c04c2dba42a29b438e9237e52537" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43123 } } peers { permanent_uuid: "20ad4e94f216486d95f9aaad1e3261d2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46171 } } peers { permanent_uuid: "651c8f05d8ce4e40a04c26c2b4296c70" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44321 } }
I20260929 15:52:50.575881 10712 catalog_manager.cc:5495] T 42ce0f50dfbd4235b15ba0c811d50c6c P 651c8f05d8ce4e40a04c26c2b4296c70 reported cstate change: term changed from 0 to 1, leader changed from <none> to 651c8f05d8ce4e40a04c26c2b4296c70 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "651c8f05d8ce4e40a04c26c2b4296c70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2891c04c2dba42a29b438e9237e52537" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20ad4e94f216486d95f9aaad1e3261d2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "651c8f05d8ce4e40a04c26c2b4296c70" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44321 } health_report { overall_health: HEALTHY } } }
I20260929 15:52:50.614584 11187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:50.608490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59786 (local address 127.0.97.1:44321)
0929 15:52:50.608685 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:50.608689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:50.608706 (+ 17us) server_negotiation.cc:407] Connection header received
0929 15:52:50.608880 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:50.608883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:50.608954 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:50.609079 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:52:50.609498 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:50.610161 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:50.611548 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:50.612101 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:50.612239 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:50.613106 (+ 867us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:52:50.613819 (+ 713us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:52:50.613985 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:50.614213 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 15:52:50.614290 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 15:52:50.622608 10991 raft_consensus.cc:1261] T 42ce0f50dfbd4235b15ba0c811d50c6c P 20ad4e94f216486d95f9aaad1e3261d2 [term 1 FOLLOWER]: Refusing update from remote peer 651c8f05d8ce4e40a04c26c2b4296c70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:52:50.622627 11125 raft_consensus.cc:1261] T 42ce0f50dfbd4235b15ba0c811d50c6c P 2891c04c2dba42a29b438e9237e52537 [term 1 FOLLOWER]: Refusing update from remote peer 651c8f05d8ce4e40a04c26c2b4296c70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:52:50.623608 11199 consensus_queue.cc:1035] T 42ce0f50dfbd4235b15ba0c811d50c6c P 651c8f05d8ce4e40a04c26c2b4296c70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2891c04c2dba42a29b438e9237e52537" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:52:50.623947 11196 consensus_queue.cc:1035] T 42ce0f50dfbd4235b15ba0c811d50c6c P 651c8f05d8ce4e40a04c26c2b4296c70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20ad4e94f216486d95f9aaad1e3261d2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:52:50.636698 11204 mvcc.cc:205] Tried to move back new op lower bound from 7334695610860814336 to 7334695610663313408. Current Snapshot: MvccSnapshot[applied={T|T < 7334695610860814336}]
I20260929 15:52:50.638916 11202 mvcc.cc:205] Tried to move back new op lower bound from 7334695610860814336 to 7334695610663313408. Current Snapshot: MvccSnapshot[applied={T|T < 7334695610860814336 or (T in {7334695610860814336})}]
I20260929 15:52:50.648610 10713 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:52:50.648939 10713 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:52:50.652798 10713 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c128f177b1ca4c55b6ca071aed1f214e: Sending DeleteTablet for 3 replicas of tablet 42ce0f50dfbd4235b15ba0c811d50c6c
I20260929 15:52:50.654203 10971 tablet_service.cc:1514] Processing DeleteTablet for tablet 42ce0f50dfbd4235b15ba0c811d50c6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:52:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50032
I20260929 15:52:50.654201 10837 tablet_service.cc:1514] Processing DeleteTablet for tablet 42ce0f50dfbd4235b15ba0c811d50c6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:52:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59770
I20260929 15:52:50.655876 11105 tablet_service.cc:1514] Processing DeleteTablet for tablet 42ce0f50dfbd4235b15ba0c811d50c6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:52:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42932
I20260929 15:52:50.657931 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 10774
I20260929 15:52:50.666663 11212 tablet_replica.cc:330] T 42ce0f50dfbd4235b15ba0c811d50c6c P 651c8f05d8ce4e40a04c26c2b4296c70: stopping tablet replica
I20260929 15:52:50.667176 11212 raft_consensus.cc:2229] T 42ce0f50dfbd4235b15ba0c811d50c6c P 651c8f05d8ce4e40a04c26c2b4296c70 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:52:50.667824 11212 raft_consensus.cc:2258] T 42ce0f50dfbd4235b15ba0c811d50c6c P 651c8f05d8ce4e40a04c26c2b4296c70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:52:50.670887 11212 ts_tablet_manager.cc:1905] T 42ce0f50dfbd4235b15ba0c811d50c6c P 651c8f05d8ce4e40a04c26c2b4296c70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:52:50.671583 11211 tablet_replica.cc:330] T 42ce0f50dfbd4235b15ba0c811d50c6c P 20ad4e94f216486d95f9aaad1e3261d2: stopping tablet replica
I20260929 15:52:50.672044 11211 raft_consensus.cc:2229] T 42ce0f50dfbd4235b15ba0c811d50c6c P 20ad4e94f216486d95f9aaad1e3261d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:52:50.672552 11211 raft_consensus.cc:2258] T 42ce0f50dfbd4235b15ba0c811d50c6c P 20ad4e94f216486d95f9aaad1e3261d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:52:50.674850 11211 ts_tablet_manager.cc:1905] T 42ce0f50dfbd4235b15ba0c811d50c6c P 20ad4e94f216486d95f9aaad1e3261d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:52:50.679190 11212 ts_tablet_manager.cc:1918] T 42ce0f50dfbd4235b15ba0c811d50c6c P 651c8f05d8ce4e40a04c26c2b4296c70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:52:50.679468 11212 log.cc:1196] T 42ce0f50dfbd4235b15ba0c811d50c6c P 651c8f05d8ce4e40a04c26c2b4296c70: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-0/wal/wals/42ce0f50dfbd4235b15ba0c811d50c6c
I20260929 15:52:50.680085 11212 ts_tablet_manager.cc:1939] T 42ce0f50dfbd4235b15ba0c811d50c6c P 651c8f05d8ce4e40a04c26c2b4296c70: Deleting consensus metadata
I20260929 15:52:50.682143 10698 catalog_manager.cc:4841] TS 651c8f05d8ce4e40a04c26c2b4296c70 (127.0.97.1:44321): tablet 42ce0f50dfbd4235b15ba0c811d50c6c (table test-table [id=da868b3576ff49ce98cee7bf6bd550a9]) successfully deleted
I20260929 15:52:50.682322 11211 ts_tablet_manager.cc:1918] T 42ce0f50dfbd4235b15ba0c811d50c6c P 20ad4e94f216486d95f9aaad1e3261d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:52:50.682476 11211 log.cc:1196] T 42ce0f50dfbd4235b15ba0c811d50c6c P 20ad4e94f216486d95f9aaad1e3261d2: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-1/wal/wals/42ce0f50dfbd4235b15ba0c811d50c6c
I20260929 15:52:50.682902 11211 ts_tablet_manager.cc:1939] T 42ce0f50dfbd4235b15ba0c811d50c6c P 20ad4e94f216486d95f9aaad1e3261d2: Deleting consensus metadata
I20260929 15:52:50.684429 10699 catalog_manager.cc:4841] TS 20ad4e94f216486d95f9aaad1e3261d2 (127.0.97.2:46171): tablet 42ce0f50dfbd4235b15ba0c811d50c6c (table test-table [id=da868b3576ff49ce98cee7bf6bd550a9]) successfully deleted
I20260929 15:52:50.685496 11213 tablet_replica.cc:330] T 42ce0f50dfbd4235b15ba0c811d50c6c P 2891c04c2dba42a29b438e9237e52537: stopping tablet replica
I20260929 15:52:50.685981 11213 raft_consensus.cc:2229] T 42ce0f50dfbd4235b15ba0c811d50c6c P 2891c04c2dba42a29b438e9237e52537 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:52:50.686496 11213 raft_consensus.cc:2258] T 42ce0f50dfbd4235b15ba0c811d50c6c P 2891c04c2dba42a29b438e9237e52537 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:52:50.688851 11213 ts_tablet_manager.cc:1905] T 42ce0f50dfbd4235b15ba0c811d50c6c P 2891c04c2dba42a29b438e9237e52537: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 15:52:50 dist-test-slave-r0ck krb5kdc[10668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 15:52:50.698235 11213 ts_tablet_manager.cc:1918] T 42ce0f50dfbd4235b15ba0c811d50c6c P 2891c04c2dba42a29b438e9237e52537: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:52:50.698444 11213 log.cc:1196] T 42ce0f50dfbd4235b15ba0c811d50c6c P 2891c04c2dba42a29b438e9237e52537: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697075657932-388-0/minicluster-data/ts-2/wal/wals/42ce0f50dfbd4235b15ba0c811d50c6c
I20260929 15:52:50.698979 11213 ts_tablet_manager.cc:1939] T 42ce0f50dfbd4235b15ba0c811d50c6c P 2891c04c2dba42a29b438e9237e52537: Deleting consensus metadata
I20260929 15:52:50.701051 10697 catalog_manager.cc:4841] TS 2891c04c2dba42a29b438e9237e52537 (127.0.97.3:43123): tablet 42ce0f50dfbd4235b15ba0c811d50c6c (table test-table [id=da868b3576ff49ce98cee7bf6bd550a9]) successfully deleted
I20260929 15:52:50.702695 11187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:50.660166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59800 (local address 127.0.97.1:44321)
0929 15:52:50.660380 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:50.660386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:50.684642 (+ 24256us) server_negotiation.cc:407] Connection header received
0929 15:52:50.684877 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:50.684881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:50.684955 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:50.685076 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:50.686537 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:50.687500 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:50.689327 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:50.689961 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:50.690131 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:50.695239 (+ 5108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:50.695258 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:50.695276 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:50.695340 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:50.699524 (+ 4184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:50.700329 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:50.700337 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:50.700340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:50.700504 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:50.700908 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:50.700918 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:50.700920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:50.701421 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:50.701785 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:50.702031 (+ 246us) server_negotiation.cc:299] Negotiation successful
0929 15:52:50.702288 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 15:52:50.703572 10897 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:52:50.810809 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 10774
I20260929 15:52:50.829330 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 10908
Sep 29 15:52:50 dist-test-slave-r0ck krb5kdc[10668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 15:52:50.846832 11185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:50.830192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50038 (local address 127.0.97.2:46171)
0929 15:52:50.830413 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:50.830418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:50.830438 (+ 20us) server_negotiation.cc:407] Connection header received
0929 15:52:50.830591 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:50.830597 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:50.830671 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:50.830774 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:50.832388 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:50.833857 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:50.835135 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:50.835539 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:50.835673 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:50.840101 (+ 4428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:50.840119 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:50.840139 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:50.840191 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:50.843464 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:50.844313 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:50.844322 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:50.844325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:50.844496 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:50.844938 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:50.844943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:50.844946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:50.845522 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:50.845799 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:50.846181 (+ 382us) server_negotiation.cc:299] Negotiation successful
0929 15:52:50.846464 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 15:52:50.847632 11031 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:52:50.939108 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 10908
I20260929 15:52:50.957136 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 11042
Sep 29 15:52:50 dist-test-slave-r0ck krb5kdc[10668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 15:52:50.971443 11184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:52:50.957992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42948 (local address 127.0.97.3:43123)
0929 15:52:50.958164 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 15:52:50.958168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:52:50.958187 (+ 19us) server_negotiation.cc:407] Connection header received
0929 15:52:50.958333 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:52:50.958336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:52:50.958399 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:52:50.958494 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:52:50.959769 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:50.960481 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:50.961786 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:52:50.962154 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:52:50.962252 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:52:50.966413 (+ 4161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:52:50.966435 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:52:50.966452 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:52:50.966492 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:52:50.969203 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:50.969867 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:50.969874 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:50.969876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:50.970006 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:52:50.970317 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:52:50.970320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:52:50.970322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:52:50.970623 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:52:50.970836 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 15:52:50.971049 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 15:52:50.971190 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 15:52:50.972110 11165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:52:51.063670 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 11042
I20260929 15:52:51.082506 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 10681
I20260929 15:52:51.084128 10743 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:52:51.194900 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 10681
2026-09-29T15:52:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4383 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:52:51 dist-test-slave-r0ck krb5kdc[11221](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:52:51 dist-test-slave-r0ck krb5kdc[11221](info): set up 2 sockets
Sep 29 15:52:51 dist-test-slave-r0ck krb5kdc[11221](info): commencing operation
krb5kdc: starting...
W20260929 15:52:52.310379 388 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:52:52 dist-test-slave-r0ck krb5kdc[11221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:52:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:52:52Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260929 15:52:52.625959 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskorfijW/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/postgres -l logfile start
2026-09-29 15:52:55.296 UTC [11251] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 15:52:55.296 UTC [11251] LOG: listening on IPv4 address "127.0.97.20", port 48857
2026-09-29 15:52:55.299 UTC [11251] LOG: listening on Unix socket "/tmp/.s.PGSQL.48857"
2026-09-29 15:52:55.326 UTC [11254] LOG: database system was shut down at 2026-09-29 15:52:53 UTC
2026-09-29 15:52:55.330 UTC [11251] LOG: database system is ready to accept connections
I20260929 15:52:56.301868 388 mini_postgres.cc:96] Postgres bound to 48857
2026-09-29 15:52:56.318 UTC [11262] FATAL: database "slave" does not exist
127.0.97.20:48857 - accepting connections
I20260929 15:52:56.319263 388 mini_ranger.cc:161] Starting Ranger...
I20260929 15:52:56.343797 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 15:52:56.820118 388 mini_ranger.cc:87] Created ranger Postgres database
I20260929 15:52:56.820235 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ranger-admin
2026-09-29 15:52:57,100 [I] DB FLAVOR :POSTGRES
2026-09-29 15:52:57,100 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 15:52:57,100 [I] Checking connection..
2026-09-29 15:52:57,100 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 15:52:57,349 [I] Checking connection passed.
2026-09-29 15:52:57,350 [I] --------- Verifying version history table ---------
2026-09-29 15:52:57,350 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 15:52:57,608 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 15:52:57,609 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 15:52:57,864 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 15:52:57,864 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 15:52:57,865 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48857/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 15:52:58.097 UTC [11326] WARNING: there is no transaction in progress
2026-09-29 15:52:58,103 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 15:52:58,103 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 15:52:58,347 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 15:52:58,347 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 15:52:58,347 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 15:52:58,592 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 15:52:58,813 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-r0ck.c.gcp-upstream.internal','N') ;"
2026-09-29 15:52:59,049 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 15:52:59,286 [I] Table x_portal_user does not exist in database ranger
2026-09-29 15:52:59,286 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 15:52:59,557 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 15:52:59,558 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 15:52:59,813 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 15:53:00,065 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 15:53:00,065 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48857/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 15:53:01.031 UTC [11434] WARNING: there is no transaction in progress
2026-09-29 15:53:01.047 UTC [11434] WARNING: there is no transaction in progress
2026-09-29 15:53:01.063 UTC [11434] WARNING: there is no transaction in progress
2026-09-29 15:53:01.078 UTC [11434] WARNING: there is no transaction in progress
2026-09-29 15:53:01.094 UTC [11434] WARNING: there is no transaction in progress
2026-09-29 15:53:01.111 UTC [11434] WARNING: there is no transaction in progress
2026-09-29 15:53:01.165 UTC [11434] WARNING: there is no transaction in progress
2026-09-29 15:53:01.171 UTC [11434] WARNING: there is no transaction in progress
2026-09-29 15:53:01.186 UTC [11434] WARNING: there is no transaction in progress
2026-09-29 15:53:01.195 UTC [11434] WARNING: there is no transaction in progress
2026-09-29 15:53:01.205 UTC [11434] WARNING: there is no transaction in progress
2026-09-29 15:53:01.219 UTC [11434] WARNING: there is no transaction in progress
2026-09-29 15:53:01.228 UTC [11434] WARNING: there is no transaction in progress
2026-09-29 15:53:01.235 UTC [11434] WARNING: there is no transaction in progress
2026-09-29 15:53:01.241 UTC [11434] WARNING: there is no transaction in progress
2026-09-29 15:53:01.249 UTC [11434] WARNING: there is no transaction in progress
2026-09-29 15:53:01,867 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 15:53:01,867 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 15:53:02,111 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 15:53:02,111 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 15:53:02,368 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 15:53:02,369 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 15:53:02,596 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 15:53:02,597 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 15:53:02,840 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 15:53:03,089 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-r0ck.c.gcp-upstream.internal';"
2026-09-29 15:53:03,324 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 15:53:03,325 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 15:53:03,547 [I] DB_PATCHES have already been applied
I20260929 15:53:03.553491 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskorfijW/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 15:53:03.553570 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260929 15:53:03.568969 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:47577
Sep 29, 2026 3:53:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 3:53:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 3:53:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 3:53:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 3:53:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 3:53:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 3:53:04 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 15:53:04.677256 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.358s user 0.008s sys 0.029s
Sep 29, 2026 3:53:24 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 3:53:24 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 3:53:24 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 3:53:24 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 3:53:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 3:53:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 3:53:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 3:53:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 3:53:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 3:53:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 3:53:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 3:53:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 3:53:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 3:53:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 3:53:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 3:53:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 3:53:24 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 3:53:25 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 15:53:26.300915 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 15:53:26 dist-test-slave-r0ck krb5kdc[11221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697206, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 15:53:26.470000 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 15:53:26.492352 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 15:53:27.402874 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 15:53:27.403056 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ranger-kms
I20260929 15:53:27.403205 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:48857
2026-09-29 15:53:27,835 [I] DB FLAVOR :POSTGRES
2026-09-29 15:53:27,835 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 15:53:27,835 [I] Checking connection
2026-09-29 15:53:27,835 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48857/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 15:53:28,089 [I] connection success
2026-09-29 15:53:28,089 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 15:53:28,089 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 15:53:28,090 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48857/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 15:53:28,339 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 15:53:28,339 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 15:53:28,339 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 15:53:28,339 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48857/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 15:53:28,592 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 15:53:29.059587 388 mini_ranger_kms.cc:310] Created kms service
I20260929 15:53:29.235420 388 mini_ranger_kms.cc:326] Created kudu user
I20260929 15:53:29.282028 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 15:53:29 dist-test-slave-r0ck krb5kdc[11221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697206, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 15:53:29.674283 388 mini_ranger_kms.cc:384] Added ranger policy
I20260929 15:53:29.674602 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 15:53:29.674642 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260929 15:53:29.691481 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11614
I20260929 15:53:29.691674 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:56747
Sep 29, 2026 3:53:31 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 3:53:31 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 3:53:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 3:53:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 3:53:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 3:53:31 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 15:53:32.009688 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.540s user 0.144s sys 0.298s
I20260929 15:53:32.009943 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 15:53:32.010164 388 mini_ranger_kms.cc:401] 127.0.97.20:56747/kms/v1/keys
Sep 29 15:53:32 dist-test-slave-r0ck krb5kdc[11221](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790697212, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 15:53:32 dist-test-slave-r0ck krb5kdc[11221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697212, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 29 15:53:32 dist-test-slave-r0ck krb5kdc[11221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697212, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 29 15:53:35 dist-test-slave-r0ck krb5kdc[11221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:53:35.186914 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35631
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35631
--ranger_config_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:56747/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:53:35.521121 11665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:53:35.521478 11665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:53:35.521656 11665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:53:35.529034 11665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:53:35.529135 11665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:53:35.529213 11665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:53:35.529295 11665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:53:35.529397 11665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:53:35.539314 11665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:56747/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35631
--ranger_config_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:53:35.540494 11665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:53:35.541795 11665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:53:35.553646 11671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:53:35.553736 11673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:53:35.553711 11670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:53:35.554869 11665 server_base.cc:1034] running on GCE node
I20260929 15:53:35.555480 11665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:53:35.556679 11665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:53:35.557871 11665 hybrid_clock.cc:648] HybridClock initialized: now 1790697215557831 us; error 31 us; skew 500 ppm
I20260929 15:53:35.558207 11665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:53:35 dist-test-slave-r0ck krb5kdc[11221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697215, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:53:35.565054 11665 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:53:35.566571 11665 webserver.cc:460] Webserver started at http://127.0.97.62:44345/ using document root <none> and password file <none>
I20260929 15:53:35.567201 11665 fs_manager.cc:359] Metadata directory not provided
I20260929 15:53:35.567343 11665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:53:35.567629 11665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 15:53:35 dist-test-slave-r0ck krb5kdc[11221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697215, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 15:53:35.628952 11665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "3a0889f548c34213a0acfad1ed35c019"
format_stamp: "Formatted at 2026-09-29 15:53:35 on dist-test-slave-r0ck"
server_key: "1d29448e02906af283eb811c0fd9a8d2"
server_key_iv: "ba13e4a1fc9ed9122d14067e9d2e8536"
server_key_version: "kuduclusterkey@0"
I20260929 15:53:35.630074 11665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "3a0889f548c34213a0acfad1ed35c019"
format_stamp: "Formatted at 2026-09-29 15:53:35 on dist-test-slave-r0ck"
server_key: "1d29448e02906af283eb811c0fd9a8d2"
server_key_iv: "ba13e4a1fc9ed9122d14067e9d2e8536"
server_key_version: "kuduclusterkey@0"
I20260929 15:53:35.636907 11665 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 15:53:35.652837 11680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:53:35.654575 11665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:53:35.654783 11665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "3a0889f548c34213a0acfad1ed35c019"
format_stamp: "Formatted at 2026-09-29 15:53:35 on dist-test-slave-r0ck"
server_key: "1d29448e02906af283eb811c0fd9a8d2"
server_key_iv: "ba13e4a1fc9ed9122d14067e9d2e8536"
server_key_version: "kuduclusterkey@0"
I20260929 15:53:35.655056 11665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:53:35.679687 11665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:53:35.683583 11665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:53:35.683815 11665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:53:35.702787 11665 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35631
I20260929 15:53:35.702816 11731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35631 every 8 connection(s)
I20260929 15:53:35.704042 11665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:53:35.707592 11732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:53:35.711478 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 11665
I20260929 15:53:35.711684 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:53:35.712448 388 external_mini_cluster.cc:1428] Setting key 37036ea428ba40d8a9c1ab3625f382f8
I20260929 15:53:35.720592 11732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a0889f548c34213a0acfad1ed35c019: Bootstrap starting.
I20260929 15:53:35.724406 11732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a0889f548c34213a0acfad1ed35c019: Neither blocks nor log segments found. Creating new log.
I20260929 15:53:35.725668 11732 log.cc:824] T 00000000000000000000000000000000 P 3a0889f548c34213a0acfad1ed35c019: Log is configured to *not* fsync() on all Append() calls
Sep 29 15:53:35 dist-test-slave-r0ck krb5kdc[11221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:53:35.728446 11732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a0889f548c34213a0acfad1ed35c019: No bootstrap required, opened a new log
I20260929 15:53:35.734687 11735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:53:35.714442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51532 (local address 127.0.97.62:35631)
0929 15:53:35.715165 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0929 15:53:35.715176 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:53:35.715214 (+ 38us) server_negotiation.cc:407] Connection header received
0929 15:53:35.716710 (+ 1496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:53:35.716731 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:53:35.717070 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:53:35.717432 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:53:35.718894 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:53:35.720122 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:53:35.721836 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:53:35.722853 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:53:35.722970 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:53:35.727602 (+ 4632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:53:35.727610 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:53:35.727621 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:53:35.727651 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:53:35.730681 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:53:35.731443 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:53:35.731452 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:53:35.731457 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:53:35.731599 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:53:35.732036 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:53:35.732040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:53:35.732046 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:53:35.732538 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:53:35.732786 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0929 15:53:35.733324 (+ 538us) server_negotiation.cc:299] Negotiation successful
0929 15:53:35.733756 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":370,"threads_started":1}
I20260929 15:53:35.737313 11732 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a0889f548c34213a0acfad1ed35c019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a0889f548c34213a0acfad1ed35c019" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35631 } }
I20260929 15:53:35.737757 11732 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a0889f548c34213a0acfad1ed35c019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:53:35.737860 11732 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a0889f548c34213a0acfad1ed35c019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a0889f548c34213a0acfad1ed35c019, State: Initialized, Role: FOLLOWER
I20260929 15:53:35.738814 11732 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a0889f548c34213a0acfad1ed35c019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a0889f548c34213a0acfad1ed35c019" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35631 } }
I20260929 15:53:35.739053 11732 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a0889f548c34213a0acfad1ed35c019 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:53:35.739197 11732 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a0889f548c34213a0acfad1ed35c019 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:53:35.739359 11732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a0889f548c34213a0acfad1ed35c019 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:53:35.741312 11732 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a0889f548c34213a0acfad1ed35c019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a0889f548c34213a0acfad1ed35c019" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35631 } }
I20260929 15:53:35.741827 11732 leader_election.cc:302] T 00000000000000000000000000000000 P 3a0889f548c34213a0acfad1ed35c019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a0889f548c34213a0acfad1ed35c019; no voters:
I20260929 15:53:35.742388 11732 leader_election.cc:288] T 00000000000000000000000000000000 P 3a0889f548c34213a0acfad1ed35c019 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:53:35.742506 11737 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a0889f548c34213a0acfad1ed35c019 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:53:35.743893 11737 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a0889f548c34213a0acfad1ed35c019 [term 1 LEADER]: Becoming Leader. State: Replica: 3a0889f548c34213a0acfad1ed35c019, State: Running, Role: LEADER
I20260929 15:53:35.744369 11737 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a0889f548c34213a0acfad1ed35c019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a0889f548c34213a0acfad1ed35c019" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35631 } }
I20260929 15:53:35.744452 11732 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a0889f548c34213a0acfad1ed35c019 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:53:35.748286 11738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a0889f548c34213a0acfad1ed35c019 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a0889f548c34213a0acfad1ed35c019" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a0889f548c34213a0acfad1ed35c019" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35631 } } }
I20260929 15:53:35.748955 11738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a0889f548c34213a0acfad1ed35c019 [sys.catalog]: This master's current role is: LEADER
I20260929 15:53:35.748432 11739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a0889f548c34213a0acfad1ed35c019 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a0889f548c34213a0acfad1ed35c019. Latest consensus state: current_term: 1 leader_uuid: "3a0889f548c34213a0acfad1ed35c019" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a0889f548c34213a0acfad1ed35c019" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35631 } } }
I20260929 15:53:35.749979 11739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a0889f548c34213a0acfad1ed35c019 [sys.catalog]: This master's current role is: LEADER
I20260929 15:53:35.751328 11745 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:53:35.756825 11745 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:53:35.757855 11732 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 15:53:35.774029 11745 catalog_manager.cc:1347] Generated new cluster ID: 6d253a40427a4204968917622af16275
I20260929 15:53:35.774183 11745 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:53:35.802666 11745 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:53:35.805287 11745 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:53:35.822489 11745 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a0889f548c34213a0acfad1ed35c019: Generated new TSK 0
I20260929 15:53:35.823348 11745 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 15:53:36.625198 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:56747/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35631
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 15:53:36 dist-test-slave-r0ck krb5kdc[11221](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790697216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 15:53:37.042994 11777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:53:37.043260 11777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:53:37.043411 11777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:53:37.051492 11777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:53:37.051651 11777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:53:37.051831 11777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:53:37.062594 11777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:56747/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:53:37.063935 11777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:53:37.065192 11777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:53:37.079458 11788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:53:37.080529 11790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:53:37.081027 11787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:53:37.081872 11777 server_base.cc:1034] running on GCE node
I20260929 15:53:37.082635 11777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:53:37.084229 11777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:53:37.085457 11777 hybrid_clock.cc:648] HybridClock initialized: now 1790697217085238 us; error 208 us; skew 500 ppm
I20260929 15:53:37.085815 11777 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:53:37 dist-test-slave-r0ck krb5kdc[11221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:53:37.090045 11777 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:53:37.093760 11777 webserver.cc:460] Webserver started at http://127.0.97.1:35633/ using document root <none> and password file <none>
I20260929 15:53:37.094655 11777 fs_manager.cc:359] Metadata directory not provided
I20260929 15:53:37.094798 11777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:53:37.095187 11777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 15:53:37 dist-test-slave-r0ck krb5kdc[11221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 15:53:37.114751 11777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-0/data/instance:
uuid: "9785c64c67404d7bb35f48d5f15f0df6"
format_stamp: "Formatted at 2026-09-29 15:53:37 on dist-test-slave-r0ck"
server_key: "caba7287a94e04626248daa824975d5b"
server_key_iv: "37acc2287051bf1388cd36aa782a7afd"
server_key_version: "kuduclusterkey@0"
I20260929 15:53:37.115852 11777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "9785c64c67404d7bb35f48d5f15f0df6"
format_stamp: "Formatted at 2026-09-29 15:53:37 on dist-test-slave-r0ck"
server_key: "caba7287a94e04626248daa824975d5b"
server_key_iv: "37acc2287051bf1388cd36aa782a7afd"
server_key_version: "kuduclusterkey@0"
I20260929 15:53:37.122968 11777 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260929 15:53:37.184197 11797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:53:37.186578 11777 fs_manager.cc:725] Time spent opening block manager: real 0.047s user 0.008s sys 0.037s
I20260929 15:53:37.186875 11777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-0/wal
uuid: "9785c64c67404d7bb35f48d5f15f0df6"
format_stamp: "Formatted at 2026-09-29 15:53:37 on dist-test-slave-r0ck"
server_key: "caba7287a94e04626248daa824975d5b"
server_key_iv: "37acc2287051bf1388cd36aa782a7afd"
server_key_version: "kuduclusterkey@0"
I20260929 15:53:37.187371 11777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:53:37.232719 11777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:53:37.238432 11777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:53:37.238859 11777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:53:37.246389 11777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:53:37.254560 11777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:53:37.254844 11777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:53:37.255050 11777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:53:37.255184 11777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 15:53:37 dist-test-slave-r0ck krb5kdc[11221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 29 15:53:37 dist-test-slave-r0ck krb5kdc[11221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 15:53:37.773427 11777 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34411
I20260929 15:53:37.773476 11911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34411 every 8 connection(s)
I20260929 15:53:37.775184 11777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:53:37.783046 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 11777
I20260929 15:53:37.783267 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:53:37.784080 388 external_mini_cluster.cc:1428] Setting key e09058ad83642e484862f0820ebd7771
Sep 29 15:53:37 dist-test-slave-r0ck krb5kdc[11221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 15:53:37.864065 11916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:53:37.811475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53217 (local address 127.0.97.62:35631)
0929 15:53:37.816949 (+ 5474us) server_negotiation.cc:206] Beginning negotiation
0929 15:53:37.816956 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:53:37.821403 (+ 4447us) server_negotiation.cc:407] Connection header received
0929 15:53:37.825666 (+ 4263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:53:37.825673 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:53:37.825775 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:53:37.825867 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:53:37.828178 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:53:37.829411 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:53:37.832938 (+ 3527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:53:37.833618 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:53:37.833721 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:53:37.846426 (+ 12705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:53:37.846437 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:53:37.846446 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:53:37.846502 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:53:37.850471 (+ 3969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:53:37.860530 (+ 10059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:53:37.860540 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:53:37.860547 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:53:37.860765 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:53:37.861539 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:53:37.861547 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:53:37.861553 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:53:37.861969 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:53:37.862124 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 15:53:37.863236 (+ 1112us) server_negotiation.cc:299] Negotiation successful
0929 15:53:37.863552 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5336,"thread_start_us":5201,"threads_started":1}
I20260929 15:53:37.866984 11915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:53:37.810721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35631 (local address 127.0.97.1:53217)
0929 15:53:37.818751 (+ 8030us) negotiation.cc:107] Waiting for socket to connect
0929 15:53:37.818804 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 15:53:37.821180 (+ 2376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:53:37.826458 (+ 5278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:53:37.826472 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:53:37.827139 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:53:37.827953 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:53:37.827981 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:53:37.829630 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:53:37.829635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:53:37.831628 (+ 1993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:53:37.831644 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:53:37.834093 (+ 2449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:53:37.834102 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:53:37.834384 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:53:37.837790 (+ 3406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:53:37.837831 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:53:37.846055 (+ 8224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:53:37.851927 (+ 5872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:53:37.851940 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:53:37.851963 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:53:37.852645 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:53:37.860972 (+ 8327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:53:37.860980 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:53:37.860987 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:53:37.861393 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:53:37.862244 (+ 851us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:53:37.862257 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:53:37.862765 (+ 508us) client_negotiation.cc:769] Sending connection context
0929 15:53:37.864702 (+ 1937us) client_negotiation.cc:240] Negotiation successful
0929 15:53:37.865403 (+ 701us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7706,"thread_start_us":6225,"threads_started":1}
I20260929 15:53:37.869395 11912 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35631
I20260929 15:53:37.869740 11912 heartbeater.cc:464] Registering TS with master...
I20260929 15:53:37.870606 11912 heartbeater.cc:511] Master 127.0.97.62:35631 requested a full tablet report, sending...
I20260929 15:53:37.872982 11697 ts_manager.cc:194] Registered new tserver with Master: 9785c64c67404d7bb35f48d5f15f0df6 (127.0.97.1:34411)
I20260929 15:53:37.874639 11697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53217
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 15:53:37.881073 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:56747/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35631
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:53:38.266387 11918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:53:38.266683 11918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:53:38.266832 11918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:53:38.274120 11918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:53:38.274221 11918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:53:38.274439 11918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:53:38.284209 11918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:56747/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:53:38.285558 11918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:53:38.286890 11918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:53:38.301369 11930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:53:38.301313 11933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:53:38.301789 11931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:53:38.302385 11918 server_base.cc:1034] running on GCE node
I20260929 15:53:38.303050 11918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:53:38.304553 11918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:53:38.305787 11918 hybrid_clock.cc:648] HybridClock initialized: now 1790697218305741 us; error 35 us; skew 500 ppm
I20260929 15:53:38.306139 11918 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:53:38 dist-test-slave-r0ck krb5kdc[11221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:53:38.309785 11918 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:53:38.311292 11918 webserver.cc:460] Webserver started at http://127.0.97.2:34265/ using document root <none> and password file <none>
I20260929 15:53:38.311947 11918 fs_manager.cc:359] Metadata directory not provided
I20260929 15:53:38.312073 11918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:53:38.312394 11918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 15:53:38 dist-test-slave-r0ck krb5kdc[11221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 15:53:38.330518 11918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-1/data/instance:
uuid: "c34d3dab5322451395667306e5bb0c27"
format_stamp: "Formatted at 2026-09-29 15:53:38 on dist-test-slave-r0ck"
server_key: "afea5e835bf8370494c97cdf97c52457"
server_key_iv: "651c798e6c5f1fd13d62fff176e5d6c4"
server_key_version: "kuduclusterkey@0"
I20260929 15:53:38.331635 11918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c34d3dab5322451395667306e5bb0c27"
format_stamp: "Formatted at 2026-09-29 15:53:38 on dist-test-slave-r0ck"
server_key: "afea5e835bf8370494c97cdf97c52457"
server_key_iv: "651c798e6c5f1fd13d62fff176e5d6c4"
server_key_version: "kuduclusterkey@0"
I20260929 15:53:38.338303 11918 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 15:53:38.355599 11940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:53:38.357297 11918 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260929 15:53:38.357563 11918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-1/wal
uuid: "c34d3dab5322451395667306e5bb0c27"
format_stamp: "Formatted at 2026-09-29 15:53:38 on dist-test-slave-r0ck"
server_key: "afea5e835bf8370494c97cdf97c52457"
server_key_iv: "651c798e6c5f1fd13d62fff176e5d6c4"
server_key_version: "kuduclusterkey@0"
I20260929 15:53:38.357810 11918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:53:38.376196 11918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:53:38.379966 11918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:53:38.380283 11918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:53:38.381140 11918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:53:38.382471 11918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:53:38.382546 11918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:53:38.382637 11918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:53:38.382682 11918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:53:38.417122 11918 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38679
I20260929 15:53:38.417125 12052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38679 every 8 connection(s)
I20260929 15:53:38.418430 11918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:53:38.421066 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 11918
I20260929 15:53:38.421265 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:53:38.421746 388 external_mini_cluster.cc:1428] Setting key 85c074a971d21d2ebee356f5bdef0e7d
Sep 29 15:53:38 dist-test-slave-r0ck krb5kdc[11221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:53:38.440007 12055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:53:38.420883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34199 (local address 127.0.97.62:35631)
0929 15:53:38.421417 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0929 15:53:38.421426 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:53:38.422495 (+ 1069us) server_negotiation.cc:407] Connection header received
0929 15:53:38.424707 (+ 2212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:53:38.424711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:53:38.424771 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:53:38.424861 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:53:38.426763 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:53:38.427472 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:53:38.429996 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:53:38.430397 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:53:38.430500 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:53:38.434425 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:53:38.434430 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:53:38.434433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:53:38.434463 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:53:38.437005 (+ 2542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:53:38.437772 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:53:38.437777 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:53:38.437785 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:53:38.437952 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:53:38.438353 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:53:38.438358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:53:38.438361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:53:38.438612 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:53:38.438764 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 15:53:38.439409 (+ 645us) server_negotiation.cc:299] Negotiation successful
0929 15:53:38.439637 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":244,"threads_started":1}
I20260929 15:53:38.441499 12056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:53:38.421448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35631 (local address 127.0.97.2:34199)
0929 15:53:38.422314 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0929 15:53:38.422376 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 15:53:38.424402 (+ 2026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:53:38.425130 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:53:38.425143 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:53:38.425812 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:53:38.426595 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:53:38.426621 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:53:38.427633 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:53:38.427641 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:53:38.429844 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:53:38.429862 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:53:38.430629 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:53:38.430638 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:53:38.430965 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:53:38.431728 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:53:38.431763 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:53:38.434210 (+ 2447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:53:38.437146 (+ 2936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:53:38.437154 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:53:38.437169 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:53:38.437644 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:53:38.438060 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:53:38.438066 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:53:38.438069 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:53:38.438235 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:53:38.438744 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:53:38.438751 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:53:38.439057 (+ 306us) client_negotiation.cc:769] Sending connection context
0929 15:53:38.439303 (+ 246us) client_negotiation.cc:240] Negotiation successful
0929 15:53:38.439794 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":365,"threads_started":1}
I20260929 15:53:38.443058 12053 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35631
I20260929 15:53:38.443359 12053 heartbeater.cc:464] Registering TS with master...
I20260929 15:53:38.444010 12053 heartbeater.cc:511] Master 127.0.97.62:35631 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 15:53:38.445612 11697 ts_manager.cc:194] Registered new tserver with Master: c34d3dab5322451395667306e5bb0c27 (127.0.97.2:38679)
I20260929 15:53:38.446467 11697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34199
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 15:53:38.473053 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:56747/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35631
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:53:38.797763 12059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:53:38.798084 12059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:53:38.798245 12059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:53:38.807166 12059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:53:38.807327 12059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:53:38.807502 12059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:53:38.817567 12059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:56747/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:53:38.818909 12059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:53:38.820235 12059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:53:38.834255 12064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:53:38.834259 12065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:53:38.834338 12067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:53:38.835186 12059 server_base.cc:1034] running on GCE node
I20260929 15:53:38.835831 12059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:53:38.836751 12059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:53:38.837975 12059 hybrid_clock.cc:648] HybridClock initialized: now 1790697218837930 us; error 57 us; skew 500 ppm
I20260929 15:53:38.838292 12059 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:53:38 dist-test-slave-r0ck krb5kdc[11221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:53:38.841805 12059 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:53:38.843102 12059 webserver.cc:460] Webserver started at http://127.0.97.3:37711/ using document root <none> and password file <none>
I20260929 15:53:38.843716 12059 fs_manager.cc:359] Metadata directory not provided
I20260929 15:53:38.843809 12059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:53:38.844044 12059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 15:53:38 dist-test-slave-r0ck krb5kdc[11221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 15:53:38.857060 12059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-2/data/instance:
uuid: "db57d760efae4ae7b48f43a765a5b5c2"
format_stamp: "Formatted at 2026-09-29 15:53:38 on dist-test-slave-r0ck"
server_key: "d6ddfb8438ba988bc4ce337963d38a98"
server_key_iv: "d1096a24b328a3736a89e0ba50d6b40f"
server_key_version: "kuduclusterkey@0"
I20260929 15:53:38.857982 12059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "db57d760efae4ae7b48f43a765a5b5c2"
format_stamp: "Formatted at 2026-09-29 15:53:38 on dist-test-slave-r0ck"
server_key: "d6ddfb8438ba988bc4ce337963d38a98"
server_key_iv: "d1096a24b328a3736a89e0ba50d6b40f"
server_key_version: "kuduclusterkey@0"
I20260929 15:53:38.863323 12059 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 15:53:38.876695 12074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:53:38.877905 11912 heartbeater.cc:503] Master 127.0.97.62:35631 was elected leader, sending a full tablet report...
I20260929 15:53:38.878295 12059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:53:38.878468 12059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-2/wal
uuid: "db57d760efae4ae7b48f43a765a5b5c2"
format_stamp: "Formatted at 2026-09-29 15:53:38 on dist-test-slave-r0ck"
server_key: "d6ddfb8438ba988bc4ce337963d38a98"
server_key_iv: "d1096a24b328a3736a89e0ba50d6b40f"
server_key_version: "kuduclusterkey@0"
I20260929 15:53:38.878729 12059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:53:38.922082 12059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:53:38.925689 12059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:53:38.925935 12059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:53:38.926786 12059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:53:38.928032 12059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:53:38.928104 12059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:53:38.928184 12059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:53:38.928226 12059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:53:38.962496 12059 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34501
I20260929 15:53:38.962554 12186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34501 every 8 connection(s)
I20260929 15:53:38.963724 12059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:53:38.973213 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 12059
I20260929 15:53:38.973421 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:53:38.973940 388 external_mini_cluster.cc:1428] Setting key fcf7d1ae1290b2a1eee4195349f9a0b2
Sep 29 15:53:38 dist-test-slave-r0ck krb5kdc[11221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:53:38.982852 12189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:53:38.966593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37007 (local address 127.0.97.62:35631)
0929 15:53:38.967077 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0929 15:53:38.967082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:53:38.967922 (+ 840us) server_negotiation.cc:407] Connection header received
0929 15:53:38.969737 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:53:38.969743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:53:38.969830 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:53:38.969950 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:53:38.971266 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:53:38.971982 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:53:38.973835 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:53:38.974295 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:53:38.974390 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:53:38.977805 (+ 3415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:53:38.977809 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:53:38.977812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:53:38.977841 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:53:38.980042 (+ 2201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:53:38.980704 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:53:38.980718 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:53:38.980721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:53:38.980834 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:53:38.981228 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:53:38.981233 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:53:38.981236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:53:38.981536 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:53:38.981693 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 15:53:38.982273 (+ 580us) server_negotiation.cc:299] Negotiation successful
0929 15:53:38.982498 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":235,"threads_started":1}
I20260929 15:53:38.983693 12190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:53:38.967114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35631 (local address 127.0.97.3:37007)
0929 15:53:38.967776 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0929 15:53:38.967811 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 15:53:38.969487 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:53:38.970164 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:53:38.970172 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:53:38.970567 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:53:38.971091 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:53:38.971106 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:53:38.972134 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:53:38.972139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:53:38.973674 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:53:38.973688 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:53:38.974500 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:53:38.974506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:53:38.974689 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:53:38.975431 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:53:38.975451 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:53:38.977603 (+ 2152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:53:38.980176 (+ 2573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:53:38.980184 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:53:38.980198 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:53:38.980579 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:53:38.980943 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:53:38.980948 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:53:38.980952 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:53:38.981110 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:53:38.981656 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:53:38.981663 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:53:38.981947 (+ 284us) client_negotiation.cc:769] Sending connection context
0929 15:53:38.982182 (+ 235us) client_negotiation.cc:240] Negotiation successful
0929 15:53:38.982644 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":273,"threads_started":1}
I20260929 15:53:38.984905 12187 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35631
I20260929 15:53:38.985133 12187 heartbeater.cc:464] Registering TS with master...
I20260929 15:53:38.986157 12187 heartbeater.cc:511] Master 127.0.97.62:35631 requested a full tablet report, sending...
I20260929 15:53:38.987489 11697 ts_manager.cc:194] Registered new tserver with Master: db57d760efae4ae7b48f43a765a5b5c2 (127.0.97.3:34501)
I20260929 15:53:38.988492 11697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:37007
I20260929 15:53:38.989180 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:53:39.005489 12189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:53:38.994129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45038 (local address 127.0.97.62:35631)
0929 15:53:38.994356 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 15:53:38.994365 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:53:38.994745 (+ 380us) server_negotiation.cc:407] Connection header received
0929 15:53:38.994862 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:53:38.994865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:53:38.994929 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:53:38.995005 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:53:38.996232 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:53:38.996876 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:53:38.998226 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:53:38.998617 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:53:38.998708 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:53:39.000928 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:53:39.000932 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:53:39.000935 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:53:39.000967 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:53:39.003312 (+ 2345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:53:39.003901 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:53:39.003905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:53:39.003907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:53:39.004017 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:53:39.004313 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:53:39.004316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:53:39.004318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:53:39.004538 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:53:39.004707 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 15:53:39.004964 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 15:53:39.005118 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 15:53:39.009794 11696 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45038:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:53:39.017299 11696 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:53:39.052701 12203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:53:39.029780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56442 (local address 127.0.97.1:34411)
0929 15:53:39.043014 (+ 13234us) server_negotiation.cc:206] Beginning negotiation
0929 15:53:39.043022 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:53:39.043049 (+ 27us) server_negotiation.cc:407] Connection header received
0929 15:53:39.043476 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:53:39.043489 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:53:39.043761 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:53:39.043968 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:53:39.045780 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:53:39.047775 (+ 1995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:53:39.050219 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:53:39.051230 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:53:39.051377 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:53:39.051627 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0929 15:53:39.052182 (+ 555us) server_negotiation.cc:299] Negotiation successful
0929 15:53:39.052341 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13077,"thread_start_us":12923,"threads_started":1}
I20260929 15:53:39.052745 12200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:53:39.029134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34411 (local address 127.0.0.1:56442)
0929 15:53:39.041230 (+ 12096us) negotiation.cc:107] Waiting for socket to connect
0929 15:53:39.041278 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 15:53:39.043169 (+ 1891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:53:39.044298 (+ 1129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:53:39.044305 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:53:39.044343 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:53:39.045562 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:53:39.045575 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:53:39.047929 (+ 2354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:53:39.047934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:53:39.050079 (+ 2145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:53:39.050088 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:53:39.051425 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:53:39.051432 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:53:39.051874 (+ 442us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:53:39.051973 (+ 99us) client_negotiation.cc:769] Sending connection context
0929 15:53:39.052119 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 15:53:39.052319 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11770,"mutex_wait_us":125,"thread_start_us":11612,"threads_started":1}
I20260929 15:53:39.053867 12201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:53:39.029215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48368 (local address 127.0.97.2:38679)
0929 15:53:39.041638 (+ 12423us) server_negotiation.cc:206] Beginning negotiation
0929 15:53:39.041651 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:53:39.042746 (+ 1095us) server_negotiation.cc:407] Connection header received
0929 15:53:39.042859 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:53:39.042871 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:53:39.043190 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:53:39.043393 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:53:39.044897 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:53:39.046653 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:53:39.049992 (+ 3339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:53:39.051439 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:53:39.051567 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:53:39.051757 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 15:53:39.053377 (+ 1620us) server_negotiation.cc:299] Negotiation successful
0929 15:53:39.053548 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12215,"thread_start_us":13307,"threads_started":1}
I20260929 15:53:39.054414 12202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:53:39.029230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38679 (local address 127.0.0.1:48368)
0929 15:53:39.042112 (+ 12882us) negotiation.cc:107] Waiting for socket to connect
0929 15:53:39.042134 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:53:39.042301 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:53:39.043771 (+ 1470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:53:39.043782 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:53:39.043822 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:53:39.044358 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:53:39.044375 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:53:39.046800 (+ 2425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:53:39.046805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:53:39.049824 (+ 3019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:53:39.049845 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:53:39.051762 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:53:39.051771 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:53:39.051927 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:53:39.051987 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 15:53:39.052103 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 15:53:39.052315 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12755,"mutex_wait_us":75,"thread_start_us":12609,"threads_started":1}
I20260929 15:53:39.056569 11845 tablet_service.cc:1467] Processing CreateTablet for tablet 0ed3f9b879944863beca649a3e674447 (DEFAULT_TABLE table=test-table [id=aaf950e4688849bc928c71618dfcc956]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:53:39.057243 11988 tablet_service.cc:1467] Processing CreateTablet for tablet 0ed3f9b879944863beca649a3e674447 (DEFAULT_TABLE table=test-table [id=aaf950e4688849bc928c71618dfcc956]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:53:39.057842 12205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:53:39.043046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34501 (local address 127.0.0.1:58324)
0929 15:53:39.046170 (+ 3124us) negotiation.cc:107] Waiting for socket to connect
0929 15:53:39.046184 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 15:53:39.046322 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:53:39.046843 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:53:39.046848 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:53:39.046876 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:53:39.047795 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:53:39.047810 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:53:39.049559 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:53:39.049564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:53:39.051868 (+ 2304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:53:39.051880 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:53:39.053847 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:53:39.053856 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:53:39.057193 (+ 3337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:53:39.057225 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 15:53:39.057326 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 15:53:39.057484 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3019,"mutex_wait_us":99,"thread_start_us":2907,"threads_started":1}
I20260929 15:53:39.057947 12204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:53:39.042617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58324 (local address 127.0.97.3:34501)
0929 15:53:39.044975 (+ 2358us) server_negotiation.cc:206] Beginning negotiation
0929 15:53:39.044986 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:53:39.046281 (+ 1295us) server_negotiation.cc:407] Connection header received
0929 15:53:39.046483 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:53:39.046488 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:53:39.046638 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:53:39.046820 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:53:39.048023 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:53:39.049433 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:53:39.052026 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:53:39.053419 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:53:39.053532 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:53:39.053726 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 15:53:39.057407 (+ 3681us) server_negotiation.cc:299] Negotiation successful
0929 15:53:39.057565 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2140,"thread_start_us":1993,"threads_started":1}
I20260929 15:53:39.057998 11845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ed3f9b879944863beca649a3e674447. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:53:39.058511 11988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ed3f9b879944863beca649a3e674447. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:53:39.061748 12122 tablet_service.cc:1467] Processing CreateTablet for tablet 0ed3f9b879944863beca649a3e674447 (DEFAULT_TABLE table=test-table [id=aaf950e4688849bc928c71618dfcc956]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:53:39.063040 12122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ed3f9b879944863beca649a3e674447. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:53:39.069967 12206 tablet_bootstrap.cc:492] T 0ed3f9b879944863beca649a3e674447 P c34d3dab5322451395667306e5bb0c27: Bootstrap starting.
I20260929 15:53:39.073161 12206 tablet_bootstrap.cc:654] T 0ed3f9b879944863beca649a3e674447 P c34d3dab5322451395667306e5bb0c27: Neither blocks nor log segments found. Creating new log.
I20260929 15:53:39.074149 12206 log.cc:824] T 0ed3f9b879944863beca649a3e674447 P c34d3dab5322451395667306e5bb0c27: Log is configured to *not* fsync() on all Append() calls
I20260929 15:53:39.077104 12207 tablet_bootstrap.cc:492] T 0ed3f9b879944863beca649a3e674447 P 9785c64c67404d7bb35f48d5f15f0df6: Bootstrap starting.
I20260929 15:53:39.078850 12209 tablet_bootstrap.cc:492] T 0ed3f9b879944863beca649a3e674447 P db57d760efae4ae7b48f43a765a5b5c2: Bootstrap starting.
I20260929 15:53:39.080971 12207 tablet_bootstrap.cc:654] T 0ed3f9b879944863beca649a3e674447 P 9785c64c67404d7bb35f48d5f15f0df6: Neither blocks nor log segments found. Creating new log.
I20260929 15:53:39.081997 12209 tablet_bootstrap.cc:654] T 0ed3f9b879944863beca649a3e674447 P db57d760efae4ae7b48f43a765a5b5c2: Neither blocks nor log segments found. Creating new log.
I20260929 15:53:39.082129 12206 tablet_bootstrap.cc:492] T 0ed3f9b879944863beca649a3e674447 P c34d3dab5322451395667306e5bb0c27: No bootstrap required, opened a new log
I20260929 15:53:39.082237 12207 log.cc:824] T 0ed3f9b879944863beca649a3e674447 P 9785c64c67404d7bb35f48d5f15f0df6: Log is configured to *not* fsync() on all Append() calls
I20260929 15:53:39.082383 12206 ts_tablet_manager.cc:1397] T 0ed3f9b879944863beca649a3e674447 P c34d3dab5322451395667306e5bb0c27: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20260929 15:53:39.083166 12209 log.cc:824] T 0ed3f9b879944863beca649a3e674447 P db57d760efae4ae7b48f43a765a5b5c2: Log is configured to *not* fsync() on all Append() calls
I20260929 15:53:39.085235 12207 tablet_bootstrap.cc:492] T 0ed3f9b879944863beca649a3e674447 P 9785c64c67404d7bb35f48d5f15f0df6: No bootstrap required, opened a new log
I20260929 15:53:39.085510 12207 ts_tablet_manager.cc:1397] T 0ed3f9b879944863beca649a3e674447 P 9785c64c67404d7bb35f48d5f15f0df6: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260929 15:53:39.085959 12209 tablet_bootstrap.cc:492] T 0ed3f9b879944863beca649a3e674447 P db57d760efae4ae7b48f43a765a5b5c2: No bootstrap required, opened a new log
I20260929 15:53:39.086190 12209 ts_tablet_manager.cc:1397] T 0ed3f9b879944863beca649a3e674447 P db57d760efae4ae7b48f43a765a5b5c2: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260929 15:53:39.092962 12206 raft_consensus.cc:348] T 0ed3f9b879944863beca649a3e674447 P c34d3dab5322451395667306e5bb0c27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34d3dab5322451395667306e5bb0c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38679 } } peers { permanent_uuid: "9785c64c67404d7bb35f48d5f15f0df6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34411 } } peers { permanent_uuid: "db57d760efae4ae7b48f43a765a5b5c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34501 } }
I20260929 15:53:39.093411 12206 raft_consensus.cc:374] T 0ed3f9b879944863beca649a3e674447 P c34d3dab5322451395667306e5bb0c27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:53:39.093503 12206 raft_consensus.cc:729] T 0ed3f9b879944863beca649a3e674447 P c34d3dab5322451395667306e5bb0c27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c34d3dab5322451395667306e5bb0c27, State: Initialized, Role: FOLLOWER
I20260929 15:53:39.094084 12209 raft_consensus.cc:348] T 0ed3f9b879944863beca649a3e674447 P db57d760efae4ae7b48f43a765a5b5c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34d3dab5322451395667306e5bb0c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38679 } } peers { permanent_uuid: "9785c64c67404d7bb35f48d5f15f0df6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34411 } } peers { permanent_uuid: "db57d760efae4ae7b48f43a765a5b5c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34501 } }
I20260929 15:53:39.094091 12206 consensus_queue.cc:260] T 0ed3f9b879944863beca649a3e674447 P c34d3dab5322451395667306e5bb0c27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34d3dab5322451395667306e5bb0c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38679 } } peers { permanent_uuid: "9785c64c67404d7bb35f48d5f15f0df6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34411 } } peers { permanent_uuid: "db57d760efae4ae7b48f43a765a5b5c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34501 } }
I20260929 15:53:39.094394 12209 raft_consensus.cc:374] T 0ed3f9b879944863beca649a3e674447 P db57d760efae4ae7b48f43a765a5b5c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:53:39.094480 12209 raft_consensus.cc:729] T 0ed3f9b879944863beca649a3e674447 P db57d760efae4ae7b48f43a765a5b5c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db57d760efae4ae7b48f43a765a5b5c2, State: Initialized, Role: FOLLOWER
I20260929 15:53:39.095064 12209 consensus_queue.cc:260] T 0ed3f9b879944863beca649a3e674447 P db57d760efae4ae7b48f43a765a5b5c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34d3dab5322451395667306e5bb0c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38679 } } peers { permanent_uuid: "9785c64c67404d7bb35f48d5f15f0df6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34411 } } peers { permanent_uuid: "db57d760efae4ae7b48f43a765a5b5c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34501 } }
I20260929 15:53:39.095206 12053 heartbeater.cc:503] Master 127.0.97.62:35631 was elected leader, sending a full tablet report...
I20260929 15:53:39.095902 12206 ts_tablet_manager.cc:1428] T 0ed3f9b879944863beca649a3e674447 P c34d3dab5322451395667306e5bb0c27: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260929 15:53:39.096549 12187 heartbeater.cc:503] Master 127.0.97.62:35631 was elected leader, sending a full tablet report...
I20260929 15:53:39.097105 12209 ts_tablet_manager.cc:1428] T 0ed3f9b879944863beca649a3e674447 P db57d760efae4ae7b48f43a765a5b5c2: Time spent starting tablet: real 0.011s user 0.008s sys 0.003s
I20260929 15:53:39.097025 12207 raft_consensus.cc:348] T 0ed3f9b879944863beca649a3e674447 P 9785c64c67404d7bb35f48d5f15f0df6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34d3dab5322451395667306e5bb0c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38679 } } peers { permanent_uuid: "9785c64c67404d7bb35f48d5f15f0df6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34411 } } peers { permanent_uuid: "db57d760efae4ae7b48f43a765a5b5c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34501 } }
I20260929 15:53:39.097432 12207 raft_consensus.cc:374] T 0ed3f9b879944863beca649a3e674447 P 9785c64c67404d7bb35f48d5f15f0df6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:53:39.097534 12207 raft_consensus.cc:729] T 0ed3f9b879944863beca649a3e674447 P 9785c64c67404d7bb35f48d5f15f0df6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9785c64c67404d7bb35f48d5f15f0df6, State: Initialized, Role: FOLLOWER
I20260929 15:53:39.098090 12207 consensus_queue.cc:260] T 0ed3f9b879944863beca649a3e674447 P 9785c64c67404d7bb35f48d5f15f0df6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34d3dab5322451395667306e5bb0c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38679 } } peers { permanent_uuid: "9785c64c67404d7bb35f48d5f15f0df6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34411 } } peers { permanent_uuid: "db57d760efae4ae7b48f43a765a5b5c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34501 } }
I20260929 15:53:39.100104 12207 ts_tablet_manager.cc:1428] T 0ed3f9b879944863beca649a3e674447 P 9785c64c67404d7bb35f48d5f15f0df6: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260929 15:53:39.130349 12214 raft_consensus.cc:482] T 0ed3f9b879944863beca649a3e674447 P 9785c64c67404d7bb35f48d5f15f0df6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:53:39.130664 12214 raft_consensus.cc:504] T 0ed3f9b879944863beca649a3e674447 P 9785c64c67404d7bb35f48d5f15f0df6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34d3dab5322451395667306e5bb0c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38679 } } peers { permanent_uuid: "9785c64c67404d7bb35f48d5f15f0df6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34411 } } peers { permanent_uuid: "db57d760efae4ae7b48f43a765a5b5c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34501 } }
I20260929 15:53:39.132290 12214 leader_election.cc:288] T 0ed3f9b879944863beca649a3e674447 P 9785c64c67404d7bb35f48d5f15f0df6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c34d3dab5322451395667306e5bb0c27 (127.0.97.2:38679), db57d760efae4ae7b48f43a765a5b5c2 (127.0.97.3:34501)
I20260929 15:53:39.139607 12201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:53:39.132663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33707 (local address 127.0.97.2:38679)
0929 15:53:39.132899 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 15:53:39.132907 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:53:39.133452 (+ 545us) server_negotiation.cc:407] Connection header received
0929 15:53:39.133730 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:53:39.133738 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:53:39.133826 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:53:39.133918 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:53:39.134542 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:53:39.135426 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:53:39.137398 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:53:39.138264 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:53:39.138361 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:53:39.138424 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 15:53:39.138836 (+ 412us) server_negotiation.cc:299] Negotiation successful
0929 15:53:39.138962 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 15:53:39.139771 12215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:53:39.132669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38679 (local address 127.0.97.1:33707)
0929 15:53:39.133315 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0929 15:53:39.133331 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:53:39.133573 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:53:39.133965 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:53:39.133974 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:53:39.134007 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:53:39.134388 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:53:39.134402 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:53:39.135559 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:53:39.135565 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:53:39.137229 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:53:39.137240 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:53:39.138466 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:53:39.138473 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:53:39.138645 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:53:39.138693 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 15:53:39.138795 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 15:53:39.138924 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":50,"thread_start_us":355,"threads_started":1}
I20260929 15:53:39.140036 12204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:53:39.133621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59559 (local address 127.0.97.3:34501)
0929 15:53:39.133872 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 15:53:39.133878 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:53:39.134234 (+ 356us) server_negotiation.cc:407] Connection header received
0929 15:53:39.134338 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:53:39.134343 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:53:39.134416 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:53:39.134503 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:53:39.135040 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:53:39.135887 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:53:39.137732 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:53:39.138616 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:53:39.138737 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:53:39.138867 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 15:53:39.139661 (+ 794us) server_negotiation.cc:299] Negotiation successful
0929 15:53:39.139793 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 15:53:39.140628 12216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:53:39.133505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34501 (local address 127.0.97.1:59559)
0929 15:53:39.134130 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0929 15:53:39.134143 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 15:53:39.134237 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:53:39.134535 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:53:39.134539 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:53:39.134560 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:53:39.134884 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:53:39.134894 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:53:39.136022 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:53:39.136025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:53:39.137596 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:53:39.137610 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:53:39.138852 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:53:39.138858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:53:39.139454 (+ 596us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:53:39.139493 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 15:53:39.140201 (+ 708us) client_negotiation.cc:240] Negotiation successful
0929 15:53:39.140329 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":46,"thread_start_us":321,"threads_started":1}
I20260929 15:53:39.140902 12008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed3f9b879944863beca649a3e674447" candidate_uuid: "9785c64c67404d7bb35f48d5f15f0df6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c34d3dab5322451395667306e5bb0c27" is_pre_election: true
I20260929 15:53:39.141485 12008 raft_consensus.cc:2454] T 0ed3f9b879944863beca649a3e674447 P c34d3dab5322451395667306e5bb0c27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9785c64c67404d7bb35f48d5f15f0df6 in term 0.
I20260929 15:53:39.141535 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed3f9b879944863beca649a3e674447" candidate_uuid: "9785c64c67404d7bb35f48d5f15f0df6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db57d760efae4ae7b48f43a765a5b5c2" is_pre_election: true
I20260929 15:53:39.142017 12142 raft_consensus.cc:2454] T 0ed3f9b879944863beca649a3e674447 P db57d760efae4ae7b48f43a765a5b5c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9785c64c67404d7bb35f48d5f15f0df6 in term 0.
I20260929 15:53:39.142088 11799 leader_election.cc:302] T 0ed3f9b879944863beca649a3e674447 P 9785c64c67404d7bb35f48d5f15f0df6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9785c64c67404d7bb35f48d5f15f0df6, c34d3dab5322451395667306e5bb0c27; no voters:
I20260929 15:53:39.142459 12214 raft_consensus.cc:2781] T 0ed3f9b879944863beca649a3e674447 P 9785c64c67404d7bb35f48d5f15f0df6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:53:39.142604 12214 raft_consensus.cc:482] T 0ed3f9b879944863beca649a3e674447 P 9785c64c67404d7bb35f48d5f15f0df6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:53:39.142709 12214 raft_consensus.cc:3037] T 0ed3f9b879944863beca649a3e674447 P 9785c64c67404d7bb35f48d5f15f0df6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:53:39.144905 12214 raft_consensus.cc:504] T 0ed3f9b879944863beca649a3e674447 P 9785c64c67404d7bb35f48d5f15f0df6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34d3dab5322451395667306e5bb0c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38679 } } peers { permanent_uuid: "9785c64c67404d7bb35f48d5f15f0df6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34411 } } peers { permanent_uuid: "db57d760efae4ae7b48f43a765a5b5c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34501 } }
I20260929 15:53:39.145514 12214 leader_election.cc:288] T 0ed3f9b879944863beca649a3e674447 P 9785c64c67404d7bb35f48d5f15f0df6 [CANDIDATE]: Term 1 election: Requested vote from peers c34d3dab5322451395667306e5bb0c27 (127.0.97.2:38679), db57d760efae4ae7b48f43a765a5b5c2 (127.0.97.3:34501)
I20260929 15:53:39.145931 12008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed3f9b879944863beca649a3e674447" candidate_uuid: "9785c64c67404d7bb35f48d5f15f0df6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c34d3dab5322451395667306e5bb0c27"
I20260929 15:53:39.146049 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed3f9b879944863beca649a3e674447" candidate_uuid: "9785c64c67404d7bb35f48d5f15f0df6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db57d760efae4ae7b48f43a765a5b5c2"
I20260929 15:53:39.146109 12008 raft_consensus.cc:3037] T 0ed3f9b879944863beca649a3e674447 P c34d3dab5322451395667306e5bb0c27 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:53:39.146251 12142 raft_consensus.cc:3037] T 0ed3f9b879944863beca649a3e674447 P db57d760efae4ae7b48f43a765a5b5c2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:53:39.148205 12008 raft_consensus.cc:2454] T 0ed3f9b879944863beca649a3e674447 P c34d3dab5322451395667306e5bb0c27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9785c64c67404d7bb35f48d5f15f0df6 in term 1.
I20260929 15:53:39.148552 11799 leader_election.cc:302] T 0ed3f9b879944863beca649a3e674447 P 9785c64c67404d7bb35f48d5f15f0df6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9785c64c67404d7bb35f48d5f15f0df6, c34d3dab5322451395667306e5bb0c27; no voters:
I20260929 15:53:39.148824 12214 raft_consensus.cc:2781] T 0ed3f9b879944863beca649a3e674447 P 9785c64c67404d7bb35f48d5f15f0df6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:53:39.148996 12142 raft_consensus.cc:2454] T 0ed3f9b879944863beca649a3e674447 P db57d760efae4ae7b48f43a765a5b5c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9785c64c67404d7bb35f48d5f15f0df6 in term 1.
I20260929 15:53:39.149326 12214 raft_consensus.cc:686] T 0ed3f9b879944863beca649a3e674447 P 9785c64c67404d7bb35f48d5f15f0df6 [term 1 LEADER]: Becoming Leader. State: Replica: 9785c64c67404d7bb35f48d5f15f0df6, State: Running, Role: LEADER
I20260929 15:53:39.150036 12214 consensus_queue.cc:237] T 0ed3f9b879944863beca649a3e674447 P 9785c64c67404d7bb35f48d5f15f0df6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34d3dab5322451395667306e5bb0c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38679 } } peers { permanent_uuid: "9785c64c67404d7bb35f48d5f15f0df6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34411 } } peers { permanent_uuid: "db57d760efae4ae7b48f43a765a5b5c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34501 } }
I20260929 15:53:39.154611 11695 catalog_manager.cc:5495] T 0ed3f9b879944863beca649a3e674447 P 9785c64c67404d7bb35f48d5f15f0df6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9785c64c67404d7bb35f48d5f15f0df6 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "9785c64c67404d7bb35f48d5f15f0df6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34d3dab5322451395667306e5bb0c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9785c64c67404d7bb35f48d5f15f0df6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db57d760efae4ae7b48f43a765a5b5c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34501 } health_report { overall_health: UNKNOWN } } }
W20260929 15:53:39.170311 12054 tablet.cc:2369] T 0ed3f9b879944863beca649a3e674447 P c34d3dab5322451395667306e5bb0c27: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:53:39.175350 12203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:53:39.170065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56446 (local address 127.0.97.1:34411)
0929 15:53:39.170264 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 15:53:39.170270 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:53:39.170288 (+ 18us) server_negotiation.cc:407] Connection header received
0929 15:53:39.170498 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:53:39.170505 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:53:39.170588 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:53:39.170740 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:53:39.171089 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:53:39.171820 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:53:39.173227 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:53:39.173686 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:53:39.173812 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:53:39.174333 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:53:39.174792 (+ 459us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:53:39.174942 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 15:53:39.175017 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 15:53:39.175093 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 15:53:39.182430 12008 raft_consensus.cc:1261] T 0ed3f9b879944863beca649a3e674447 P c34d3dab5322451395667306e5bb0c27 [term 1 FOLLOWER]: Refusing update from remote peer 9785c64c67404d7bb35f48d5f15f0df6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:53:39.182495 12142 raft_consensus.cc:1261] T 0ed3f9b879944863beca649a3e674447 P db57d760efae4ae7b48f43a765a5b5c2 [term 1 FOLLOWER]: Refusing update from remote peer 9785c64c67404d7bb35f48d5f15f0df6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:53:39.183331 12217 consensus_queue.cc:1035] T 0ed3f9b879944863beca649a3e674447 P 9785c64c67404d7bb35f48d5f15f0df6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c34d3dab5322451395667306e5bb0c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 15:53:39.184111 12214 consensus_queue.cc:1035] T 0ed3f9b879944863beca649a3e674447 P 9785c64c67404d7bb35f48d5f15f0df6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db57d760efae4ae7b48f43a765a5b5c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:53:39.195633 12221 mvcc.cc:205] Tried to move back new op lower bound from 7334695809762566144 to 7334695809642377216. Current Snapshot: MvccSnapshot[applied={T|T < 7334695809762566144}]
I20260929 15:53:39.206966 11697 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:53:39.207195 11697 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:53:39.210319 11697 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3a0889f548c34213a0acfad1ed35c019: Sending DeleteTablet for 3 replicas of tablet 0ed3f9b879944863beca649a3e674447
I20260929 15:53:39.211377 11988 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ed3f9b879944863beca649a3e674447 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:53:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48368
I20260929 15:53:39.211421 11845 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ed3f9b879944863beca649a3e674447 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:53:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56442
I20260929 15:53:39.211616 12122 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ed3f9b879944863beca649a3e674447 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:53:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58324
I20260929 15:53:39.214251 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 11777
I20260929 15:53:39.229066 12229 tablet_replica.cc:330] T 0ed3f9b879944863beca649a3e674447 P db57d760efae4ae7b48f43a765a5b5c2: stopping tablet replica
I20260929 15:53:39.229524 12229 raft_consensus.cc:2229] T 0ed3f9b879944863beca649a3e674447 P db57d760efae4ae7b48f43a765a5b5c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:53:39.230024 12229 raft_consensus.cc:2258] T 0ed3f9b879944863beca649a3e674447 P db57d760efae4ae7b48f43a765a5b5c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:53:39.236495 12230 tablet_replica.cc:330] T 0ed3f9b879944863beca649a3e674447 P 9785c64c67404d7bb35f48d5f15f0df6: stopping tablet replica
I20260929 15:53:39.236820 12231 tablet_replica.cc:330] T 0ed3f9b879944863beca649a3e674447 P c34d3dab5322451395667306e5bb0c27: stopping tablet replica
I20260929 15:53:39.236929 12230 raft_consensus.cc:2229] T 0ed3f9b879944863beca649a3e674447 P 9785c64c67404d7bb35f48d5f15f0df6 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:53:39.237190 12231 raft_consensus.cc:2229] T 0ed3f9b879944863beca649a3e674447 P c34d3dab5322451395667306e5bb0c27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:53:39.237586 12230 raft_consensus.cc:2258] T 0ed3f9b879944863beca649a3e674447 P 9785c64c67404d7bb35f48d5f15f0df6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:53:39.237669 12231 raft_consensus.cc:2258] T 0ed3f9b879944863beca649a3e674447 P c34d3dab5322451395667306e5bb0c27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:53:39.239681 12231 ts_tablet_manager.cc:1905] T 0ed3f9b879944863beca649a3e674447 P c34d3dab5322451395667306e5bb0c27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:53:39.239681 12229 ts_tablet_manager.cc:1905] T 0ed3f9b879944863beca649a3e674447 P db57d760efae4ae7b48f43a765a5b5c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:53:39.240159 12230 ts_tablet_manager.cc:1905] T 0ed3f9b879944863beca649a3e674447 P 9785c64c67404d7bb35f48d5f15f0df6: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 15:53:39 dist-test-slave-r0ck krb5kdc[11221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 15:53:39.246466 12229 ts_tablet_manager.cc:1918] T 0ed3f9b879944863beca649a3e674447 P db57d760efae4ae7b48f43a765a5b5c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:53:39.246628 12229 log.cc:1196] T 0ed3f9b879944863beca649a3e674447 P db57d760efae4ae7b48f43a765a5b5c2: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-2/wal/wals/0ed3f9b879944863beca649a3e674447
I20260929 15:53:39.247129 12229 ts_tablet_manager.cc:1939] T 0ed3f9b879944863beca649a3e674447 P db57d760efae4ae7b48f43a765a5b5c2: Deleting consensus metadata
I20260929 15:53:39.247469 12231 ts_tablet_manager.cc:1918] T 0ed3f9b879944863beca649a3e674447 P c34d3dab5322451395667306e5bb0c27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:53:39.248044 12230 ts_tablet_manager.cc:1918] T 0ed3f9b879944863beca649a3e674447 P 9785c64c67404d7bb35f48d5f15f0df6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:53:39.248186 12230 log.cc:1196] T 0ed3f9b879944863beca649a3e674447 P 9785c64c67404d7bb35f48d5f15f0df6: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-0/wal/wals/0ed3f9b879944863beca649a3e674447
I20260929 15:53:39.248304 12231 log.cc:1196] T 0ed3f9b879944863beca649a3e674447 P c34d3dab5322451395667306e5bb0c27: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697075657932-388-0/minicluster-data/ts-1/wal/wals/0ed3f9b879944863beca649a3e674447
I20260929 15:53:39.248721 12230 ts_tablet_manager.cc:1939] T 0ed3f9b879944863beca649a3e674447 P 9785c64c67404d7bb35f48d5f15f0df6: Deleting consensus metadata
I20260929 15:53:39.248975 11682 catalog_manager.cc:4841] TS db57d760efae4ae7b48f43a765a5b5c2 (127.0.97.3:34501): tablet 0ed3f9b879944863beca649a3e674447 (table test-table [id=aaf950e4688849bc928c71618dfcc956]) successfully deleted
I20260929 15:53:39.249482 12231 ts_tablet_manager.cc:1939] T 0ed3f9b879944863beca649a3e674447 P c34d3dab5322451395667306e5bb0c27: Deleting consensus metadata
I20260929 15:53:39.250308 11683 catalog_manager.cc:4841] TS 9785c64c67404d7bb35f48d5f15f0df6 (127.0.97.1:34411): tablet 0ed3f9b879944863beca649a3e674447 (table test-table [id=aaf950e4688849bc928c71618dfcc956]) successfully deleted
I20260929 15:53:39.251113 11682 catalog_manager.cc:4841] TS c34d3dab5322451395667306e5bb0c27 (127.0.97.2:38679): tablet 0ed3f9b879944863beca649a3e674447 (table test-table [id=aaf950e4688849bc928c71618dfcc956]) successfully deleted
I20260929 15:53:39.251503 12203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:53:39.215654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56456 (local address 127.0.97.1:34411)
0929 15:53:39.215823 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 15:53:39.215829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:53:39.230720 (+ 14891us) server_negotiation.cc:407] Connection header received
0929 15:53:39.230986 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:53:39.230992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:53:39.231098 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:53:39.231169 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:53:39.232923 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:53:39.234178 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:53:39.237484 (+ 3306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:53:39.238179 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:53:39.238295 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:53:39.243638 (+ 5343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:53:39.243653 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:53:39.243672 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:53:39.243713 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:53:39.248045 (+ 4332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:53:39.249000 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:53:39.249008 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:53:39.249011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:53:39.249188 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:53:39.249652 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:53:39.249657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:53:39.249660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:53:39.250069 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:53:39.250380 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0929 15:53:39.250926 (+ 546us) server_negotiation.cc:299] Negotiation successful
0929 15:53:39.251156 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 15:53:39.252275 11907 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:53:39.347731 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 11777
I20260929 15:53:39.366034 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 11918
Sep 29 15:53:39 dist-test-slave-r0ck krb5kdc[11221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 15:53:39.380333 12201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:53:39.366780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48382 (local address 127.0.97.2:38679)
0929 15:53:39.366942 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 15:53:39.366946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:53:39.366962 (+ 16us) server_negotiation.cc:407] Connection header received
0929 15:53:39.367120 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:53:39.367124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:53:39.367194 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:53:39.367280 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:53:39.368441 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:53:39.369227 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:53:39.370516 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:53:39.370882 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:53:39.370972 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:53:39.375058 (+ 4086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:53:39.375069 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:53:39.375084 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:53:39.375111 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:53:39.377750 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:53:39.378387 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:53:39.378399 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:53:39.378401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:53:39.378513 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:53:39.378868 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:53:39.378875 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:53:39.378878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:53:39.379223 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:53:39.379505 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0929 15:53:39.379826 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 15:53:39.380029 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 15:53:39.380925 12048 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:53:39.468210 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 11918
I20260929 15:53:39.484869 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 12059
Sep 29 15:53:39 dist-test-slave-r0ck krb5kdc[11221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 15:53:39.499235 12204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:53:39.485671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58330 (local address 127.0.97.3:34501)
0929 15:53:39.485841 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 15:53:39.485846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:53:39.485879 (+ 33us) server_negotiation.cc:407] Connection header received
0929 15:53:39.486040 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:53:39.486043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:53:39.486120 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:53:39.486207 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:53:39.487385 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:53:39.488123 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:53:39.489529 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:53:39.489906 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:53:39.489998 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:53:39.494209 (+ 4211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:53:39.494222 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:53:39.494237 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:53:39.494266 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:53:39.496896 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:53:39.497588 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:53:39.497597 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:53:39.497600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:53:39.497721 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:53:39.498025 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:53:39.498028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:53:39.498030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:53:39.498370 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:53:39.498580 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 15:53:39.498806 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 15:53:39.498971 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 15:53:39.499840 12182 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:53:39.605516 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 12059
I20260929 15:53:39.622177 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 11665
I20260929 15:53:39.623301 11727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:53:39.733395 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 11665
I20260929 15:53:39.755474 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 15:53:40.090484 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 15:53:40.090718 388 mini_ranger.cc:66] Stopping Ranger...
I20260929 15:53:40.435837 388 mini_ranger.cc:68] Stopped Ranger
2026-09-29 15:53:40.436 UTC [11251] LOG: received smart shutdown request
2026-09-29 15:53:40.439 UTC [11251] LOG: background worker "logical replication launcher" (PID 11260) exited with exit code 1
2026-09-29 15:53:40.439 UTC [11255] LOG: shutting down
2026-09-29 15:53:40.626 UTC [11251] LOG: database system is shut down
2026-09-29T15:53:40Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49549 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:53:40 dist-test-slave-r0ck krb5kdc[12245](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:53:40 dist-test-slave-r0ck krb5kdc[12245](info): set up 2 sockets
Sep 29 15:53:40 dist-test-slave-r0ck krb5kdc[12245](info): commencing operation
krb5kdc: starting...
W20260929 15:53:41.861691 388 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:53:42 dist-test-slave-r0ck krb5kdc[12245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:53:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:53:42Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260929 15:53:42.189730 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskorfijW/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/postgres -l logfile start
2026-09-29 15:53:44.827 UTC [12275] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 15:53:44.827 UTC [12275] LOG: listening on IPv4 address "127.0.97.20", port 54181
2026-09-29 15:53:44.830 UTC [12275] LOG: listening on Unix socket "/tmp/.s.PGSQL.54181"
2026-09-29 15:53:44.854 UTC [12278] LOG: database system was shut down at 2026-09-29 15:53:43 UTC
2026-09-29 15:53:44.857 UTC [12275] LOG: database system is ready to accept connections
I20260929 15:53:45.845294 388 mini_postgres.cc:96] Postgres bound to 54181
2026-09-29 15:53:45.862 UTC [12286] FATAL: database "slave" does not exist
127.0.97.20:54181 - accepting connections
I20260929 15:53:45.862886 388 mini_ranger.cc:161] Starting Ranger...
I20260929 15:53:45.888855 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 15:53:46.423048 388 mini_ranger.cc:87] Created ranger Postgres database
I20260929 15:53:46.423174 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ranger-admin
2026-09-29 15:53:46,720 [I] DB FLAVOR :POSTGRES
2026-09-29 15:53:46,720 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 15:53:46,720 [I] Checking connection..
2026-09-29 15:53:46,721 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54181/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 15:53:46,983 [I] Checking connection passed.
2026-09-29 15:53:46,983 [I] --------- Verifying version history table ---------
2026-09-29 15:53:46,983 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54181/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 15:53:47,245 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 15:53:47,245 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54181/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 15:53:47,491 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 15:53:47,491 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 15:53:47,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54181/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 15:53:47.744 UTC [12350] WARNING: there is no transaction in progress
2026-09-29 15:53:47,750 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 15:53:47,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54181/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 15:53:48,017 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 15:53:48,018 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 15:53:48,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54181/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 15:53:48,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54181/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 15:53:48,497 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54181/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-r0ck.c.gcp-upstream.internal','N') ;"
2026-09-29 15:53:48,745 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54181/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 15:53:49,004 [I] Table x_portal_user does not exist in database ranger
2026-09-29 15:53:49,004 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54181/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 15:53:49,253 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 15:53:49,254 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54181/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 15:53:49,504 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54181/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 15:53:49,748 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 15:53:49,748 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54181/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 15:53:50.746 UTC [12458] WARNING: there is no transaction in progress
2026-09-29 15:53:50.763 UTC [12458] WARNING: there is no transaction in progress
2026-09-29 15:53:50.779 UTC [12458] WARNING: there is no transaction in progress
2026-09-29 15:53:50.795 UTC [12458] WARNING: there is no transaction in progress
2026-09-29 15:53:50.811 UTC [12458] WARNING: there is no transaction in progress
2026-09-29 15:53:50.831 UTC [12458] WARNING: there is no transaction in progress
2026-09-29 15:53:50.889 UTC [12458] WARNING: there is no transaction in progress
2026-09-29 15:53:50.896 UTC [12458] WARNING: there is no transaction in progress
2026-09-29 15:53:50.911 UTC [12458] WARNING: there is no transaction in progress
2026-09-29 15:53:50.921 UTC [12458] WARNING: there is no transaction in progress
2026-09-29 15:53:50.931 UTC [12458] WARNING: there is no transaction in progress
2026-09-29 15:53:50.945 UTC [12458] WARNING: there is no transaction in progress
2026-09-29 15:53:50.954 UTC [12458] WARNING: there is no transaction in progress
2026-09-29 15:53:50.961 UTC [12458] WARNING: there is no transaction in progress
2026-09-29 15:53:50.968 UTC [12458] WARNING: there is no transaction in progress
2026-09-29 15:53:50.976 UTC [12458] WARNING: there is no transaction in progress
2026-09-29 15:53:51,616 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 15:53:51,616 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54181/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 15:53:51,866 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 15:53:51,866 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54181/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 15:53:52,119 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 15:53:52,120 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54181/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 15:53:52,366 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 15:53:52,367 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54181/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 15:53:52,593 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54181/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 15:53:52,840 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54181/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-r0ck.c.gcp-upstream.internal';"
2026-09-29 15:53:53,067 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 15:53:53,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54181/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 15:53:53,291 [I] DB_PATCHES have already been applied
I20260929 15:53:53.297683 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskorfijW/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 15:53:53.297792 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260929 15:53:53.311682 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:42345
Sep 29, 2026 3:53:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 3:53:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 3:53:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 3:53:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 3:53:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 3:53:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 3:53:54 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 15:53:54.454877 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.592s user 0.000s sys 0.036s
Sep 29, 2026 3:54:13 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 3:54:13 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 29, 2026 3:54:13 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 3:54:13 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 3:54:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 3:54:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 3:54:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 3:54:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 3:54:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 3:54:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 3:54:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 3:54:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 3:54:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 3:54:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 3:54:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 3:54:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 3:54:13 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 3:54:14 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 15:54:15.102377 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 15:54:15 dist-test-slave-r0ck krb5kdc[12245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697255, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 15:54:15.260663 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 15:54:15.280635 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 15:54:16.153250 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 15:54:16.153394 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ranger-kms
I20260929 15:54:16.153450 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:54181
2026-09-29 15:54:16,501 [I] DB FLAVOR :POSTGRES
2026-09-29 15:54:16,501 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 15:54:16,501 [I] Checking connection
2026-09-29 15:54:16,502 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54181/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 15:54:16,741 [I] connection success
2026-09-29 15:54:16,742 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 15:54:16,742 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 15:54:16,742 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54181/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 15:54:16,974 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 15:54:16,974 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 15:54:16,974 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 15:54:16,974 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54181/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 15:54:17,252 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 15:54:17.702412 388 mini_ranger_kms.cc:310] Created kms service
I20260929 15:54:17.869655 388 mini_ranger_kms.cc:326] Created kudu user
I20260929 15:54:17.915902 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 15:54:17 dist-test-slave-r0ck krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697255, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 15:54:18.305724 388 mini_ranger_kms.cc:384] Added ranger policy
I20260929 15:54:18.306035 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskorfijW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskorfijW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 15:54:18.306095 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260929 15:54:18.320680 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12638
I20260929 15:54:18.320809 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:40351
Sep 29, 2026 3:54:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 3:54:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 3:54:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 3:54:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 3:54:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 3:54:19 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 29 15:54:20 dist-test-slave-r0ck krb5kdc[12245](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790697260, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:54:20.633800 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.373s user 0.068s sys 0.293s
I20260929 15:54:20.634061 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 15:54:20.634227 388 mini_ranger_kms.cc:401] 127.0.97.20:40351/kms/v1/keys
Sep 29 15:54:21 dist-test-slave-r0ck krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697260, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 29 15:54:21 dist-test-slave-r0ck krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697260, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 29 15:54:23 dist-test-slave-r0ck krb5kdc[12245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:54:23.572535 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42957
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42957
--ranger_config_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:40351/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:54:23.924615 12689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:54:23.924887 12689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:54:23.924973 12689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:54:23.932010 12689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:54:23.932108 12689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:54:23.932206 12689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:54:23.932284 12689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:54:23.932355 12689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:54:23.941939 12689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:40351/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42957
--ranger_config_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:54:23.943094 12689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:54:23.944321 12689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:54:23.955977 12695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:23.956063 12694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:23.956017 12697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:54:23.958662 12689 server_base.cc:1034] running on GCE node
I20260929 15:54:23.959432 12689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:54:23.960736 12689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:54:23.961962 12689 hybrid_clock.cc:648] HybridClock initialized: now 1790697263961933 us; error 49 us; skew 500 ppm
I20260929 15:54:23.962314 12689 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:54:23 dist-test-slave-r0ck krb5kdc[12245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:54:23.965248 12689 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:54:23.966828 12689 webserver.cc:460] Webserver started at http://127.0.97.62:45235/ using document root <none> and password file <none>
I20260929 15:54:23.967489 12689 fs_manager.cc:359] Metadata directory not provided
I20260929 15:54:23.967633 12689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:54:23.967933 12689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 15:54:23 dist-test-slave-r0ck krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 15:54:24.033800 12689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "576b6935026a416e9d336bd069ec551e"
format_stamp: "Formatted at 2026-09-29 15:54:24 on dist-test-slave-r0ck"
server_key: "681f188b63102d291ec61903d2243310"
server_key_iv: "f9572bcbedef15a3577fc903746c4c38"
server_key_version: "kuduclusterkey@0"
I20260929 15:54:24.034628 12689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "576b6935026a416e9d336bd069ec551e"
format_stamp: "Formatted at 2026-09-29 15:54:24 on dist-test-slave-r0ck"
server_key: "681f188b63102d291ec61903d2243310"
server_key_iv: "f9572bcbedef15a3577fc903746c4c38"
server_key_version: "kuduclusterkey@0"
I20260929 15:54:24.039912 12689 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 15:54:24.059758 12704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:24.061278 12689 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 15:54:24.061512 12689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "576b6935026a416e9d336bd069ec551e"
format_stamp: "Formatted at 2026-09-29 15:54:24 on dist-test-slave-r0ck"
server_key: "681f188b63102d291ec61903d2243310"
server_key_iv: "f9572bcbedef15a3577fc903746c4c38"
server_key_version: "kuduclusterkey@0"
I20260929 15:54:24.061738 12689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:54:24.082710 12689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:54:24.086694 12689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:54:24.087016 12689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:54:24.106729 12689 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42957
I20260929 15:54:24.106751 12755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42957 every 8 connection(s)
I20260929 15:54:24.108107 12689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:54:24.111924 12756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:54:24.118479 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 12689
I20260929 15:54:24.118701 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:54:24.119498 388 external_mini_cluster.cc:1428] Setting key 423532a1493a070334ec3329f80e193a
I20260929 15:54:24.123528 12756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e: Bootstrap starting.
I20260929 15:54:24.128455 12756 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e: Neither blocks nor log segments found. Creating new log.
I20260929 15:54:24.130187 12756 log.cc:824] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e: Log is configured to *not* fsync() on all Append() calls
I20260929 15:54:24.139394 12756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e: No bootstrap required, opened a new log
I20260929 15:54:24.148444 12756 raft_consensus.cc:348] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "576b6935026a416e9d336bd069ec551e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42957 } }
I20260929 15:54:24.148831 12756 raft_consensus.cc:374] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:54:24.148968 12756 raft_consensus.cc:729] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 576b6935026a416e9d336bd069ec551e, State: Initialized, Role: FOLLOWER
I20260929 15:54:24.149678 12756 consensus_queue.cc:260] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "576b6935026a416e9d336bd069ec551e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42957 } }
Sep 29 15:54:24 dist-test-slave-r0ck krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:54:24.150008 12756 raft_consensus.cc:388] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:54:24.150137 12756 raft_consensus.cc:482] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:54:24.150326 12756 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:54:24.153548 12756 raft_consensus.cc:504] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "576b6935026a416e9d336bd069ec551e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42957 } }
I20260929 15:54:24.154184 12756 leader_election.cc:302] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 576b6935026a416e9d336bd069ec551e; no voters:
I20260929 15:54:24.154856 12756 leader_election.cc:288] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:54:24.155058 12761 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:54:24.162147 12761 raft_consensus.cc:686] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e [term 1 LEADER]: Becoming Leader. State: Replica: 576b6935026a416e9d336bd069ec551e, State: Running, Role: LEADER
I20260929 15:54:24.162863 12759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:24.121615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47140 (local address 127.0.97.62:42957)
0929 15:54:24.136460 (+ 14845us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:24.136480 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:24.136536 (+ 56us) server_negotiation.cc:407] Connection header received
0929 15:54:24.138864 (+ 2328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:24.138890 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:24.139353 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:24.139791 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:24.141389 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:24.143443 (+ 2054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:24.145200 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:24.146270 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:24.146455 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:24.151214 (+ 4759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:24.151224 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:24.151243 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:24.151294 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:24.155402 (+ 4108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:24.156541 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:24.156554 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:24.156567 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:24.156781 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:24.157367 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:24.157375 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:24.157381 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:24.158299 (+ 918us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:24.160081 (+ 1782us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:24.160444 (+ 363us) server_negotiation.cc:299] Negotiation successful
0929 15:54:24.161035 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14533,"thread_start_us":14710,"threads_started":1}
I20260929 15:54:24.162839 12761 consensus_queue.cc:237] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "576b6935026a416e9d336bd069ec551e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42957 } }
I20260929 15:54:24.166625 12756 sys_catalog.cc:564] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:54:24.172932 12762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 576b6935026a416e9d336bd069ec551e. Latest consensus state: current_term: 1 leader_uuid: "576b6935026a416e9d336bd069ec551e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "576b6935026a416e9d336bd069ec551e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42957 } } }
I20260929 15:54:24.173300 12762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e [sys.catalog]: This master's current role is: LEADER
I20260929 15:54:24.174331 12767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:54:24.174182 12763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "576b6935026a416e9d336bd069ec551e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "576b6935026a416e9d336bd069ec551e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42957 } } }
I20260929 15:54:24.174629 12763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e [sys.catalog]: This master's current role is: LEADER
I20260929 15:54:24.187497 12767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:54:24.205514 12767 catalog_manager.cc:1347] Generated new cluster ID: 67d95d0d91604039b55cb632a146d2eb
I20260929 15:54:24.205629 12767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:54:24.211995 12756 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 15:54:24.233373 12767 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:54:24.235371 12767 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:54:24.252141 12767 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e: Generated new TSK 0
I20260929 15:54:24.253216 12767 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 15:54:24.942481 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:40351/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42957
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 15:54:25 dist-test-slave-r0ck krb5kdc[12245](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790697265, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 15:54:25.346232 12801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:54:25.346771 12801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:54:25.346941 12801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:54:25.359036 12801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:54:25.359185 12801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:54:25.359373 12801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:54:25.375074 12801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:40351/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:54:25.376832 12801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:54:25.378547 12801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:54:25.396220 12811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:25.397073 12812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:25.414090 12814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:54:25.414708 12801 server_base.cc:1034] running on GCE node
I20260929 15:54:25.415571 12801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:54:25.417640 12801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:54:25.418965 12801 hybrid_clock.cc:648] HybridClock initialized: now 1790697265418891 us; error 73 us; skew 500 ppm
I20260929 15:54:25.419409 12801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:54:25 dist-test-slave-r0ck krb5kdc[12245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697265, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:54:25.426877 12801 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:54:25.430197 12801 webserver.cc:460] Webserver started at http://127.0.97.1:46653/ using document root <none> and password file <none>
I20260929 15:54:25.431200 12801 fs_manager.cc:359] Metadata directory not provided
I20260929 15:54:25.431411 12801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:54:25.431836 12801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 15:54:25 dist-test-slave-r0ck krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697265, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 15:54:25.453868 12801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-0/data/instance:
uuid: "fe46a8f385dd458f8992090bf3a66bb7"
format_stamp: "Formatted at 2026-09-29 15:54:25 on dist-test-slave-r0ck"
server_key: "f63cc0bdbd40eb9f46ba78358ae17209"
server_key_iv: "1fbaa2faa397f63375df7c04963d95d3"
server_key_version: "kuduclusterkey@0"
I20260929 15:54:25.454934 12801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "fe46a8f385dd458f8992090bf3a66bb7"
format_stamp: "Formatted at 2026-09-29 15:54:25 on dist-test-slave-r0ck"
server_key: "f63cc0bdbd40eb9f46ba78358ae17209"
server_key_iv: "1fbaa2faa397f63375df7c04963d95d3"
server_key_version: "kuduclusterkey@0"
I20260929 15:54:25.469669 12801 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.003s sys 0.009s
I20260929 15:54:25.502012 12821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:25.506999 12801 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.005s sys 0.007s
I20260929 15:54:25.507330 12801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-0/wal
uuid: "fe46a8f385dd458f8992090bf3a66bb7"
format_stamp: "Formatted at 2026-09-29 15:54:25 on dist-test-slave-r0ck"
server_key: "f63cc0bdbd40eb9f46ba78358ae17209"
server_key_iv: "1fbaa2faa397f63375df7c04963d95d3"
server_key_version: "kuduclusterkey@0"
I20260929 15:54:25.507786 12801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:54:25.554203 12801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:54:25.559405 12801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:54:25.559779 12801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:54:25.565510 12801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:54:25.570122 12801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:54:25.570230 12801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:25.570370 12801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:54:25.570454 12801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:25.772750 12801 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38189
I20260929 15:54:25.772771 12933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38189 every 8 connection(s)
I20260929 15:54:25.774559 12801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:54:25.782658 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 12801
I20260929 15:54:25.782909 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:54:25.783907 388 external_mini_cluster.cc:1428] Setting key dc16ea97976ac1b56c90521fa0cb5823
Sep 29 15:54:25 dist-test-slave-r0ck krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697265, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:54:25.820502 12937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:25.778451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42957 (local address 127.0.97.1:52713)
0929 15:54:25.791409 (+ 12958us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:25.791465 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 15:54:25.793875 (+ 2410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:54:25.797904 (+ 4029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:54:25.797919 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:54:25.798591 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:54:25.799473 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:25.799500 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:25.801154 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:25.801159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:25.803497 (+ 2338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:25.803525 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:25.804952 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:25.804962 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:25.805285 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:25.807602 (+ 2317us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:54:25.807668 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:54:25.811069 (+ 3401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:54:25.815454 (+ 4385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:25.815466 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:25.815488 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:25.816146 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:25.816609 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:25.816616 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:25.816623 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:25.816879 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:25.817620 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:54:25.817631 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:54:25.818149 (+ 518us) client_negotiation.cc:769] Sending connection context
0929 15:54:25.818453 (+ 304us) client_negotiation.cc:240] Negotiation successful
0929 15:54:25.819028 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12650,"thread_start_us":12408,"threads_started":1}
I20260929 15:54:25.821950 12936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:25.777869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52713 (local address 127.0.97.62:42957)
0929 15:54:25.792436 (+ 14567us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:25.792442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:25.792461 (+ 19us) server_negotiation.cc:407] Connection header received
0929 15:54:25.794265 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:25.794270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:25.794361 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:25.794455 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:25.799755 (+ 5300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:25.800956 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:25.803722 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:25.804489 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:25.804628 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:25.811335 (+ 6707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:25.811344 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:25.811352 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:25.811408 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:25.815286 (+ 3878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:25.816294 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:25.816302 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:25.816309 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:25.816469 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:25.817012 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:25.817019 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:25.817025 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:25.817414 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:25.817553 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:25.821198 (+ 3645us) server_negotiation.cc:299] Negotiation successful
0929 15:54:25.821484 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14442,"thread_start_us":14337,"threads_started":1}
I20260929 15:54:25.824211 12934 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42957
I20260929 15:54:25.824528 12934 heartbeater.cc:464] Registering TS with master...
I20260929 15:54:25.825232 12934 heartbeater.cc:511] Master 127.0.97.62:42957 requested a full tablet report, sending...
Sep 29 15:54:25 dist-test-slave-r0ck krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697265, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 15:54:25.827662 12721 ts_manager.cc:194] Registered new tserver with Master: fe46a8f385dd458f8992090bf3a66bb7 (127.0.97.1:38189)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 15:54:25.829545 12721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52713
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 15:54:25.872908 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:40351/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42957
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 15:54:26 dist-test-slave-r0ck krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697265, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260929 15:54:26.288216 12940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:54:26.288699 12940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:54:26.288882 12940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:54:26.299330 12940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:54:26.299551 12940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:54:26.299783 12940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:54:26.315127 12940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:40351/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:54:26.317109 12940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:54:26.318867 12940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:54:26.353776 12948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:26.353758 12947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:54:26.359905 12940 server_base.cc:1034] running on GCE node
W20260929 15:54:26.360242 12950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:54:26.360950 12940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:54:26.362083 12940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:54:26.363358 12940 hybrid_clock.cc:648] HybridClock initialized: now 1790697266363294 us; error 55 us; skew 500 ppm
I20260929 15:54:26.363708 12940 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:54:26 dist-test-slave-r0ck krb5kdc[12245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:54:26.367383 12940 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:54:26.394438 12940 webserver.cc:460] Webserver started at http://127.0.97.2:45595/ using document root <none> and password file <none>
I20260929 15:54:26.395308 12940 fs_manager.cc:359] Metadata directory not provided
I20260929 15:54:26.395449 12940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:54:26.395779 12940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 15:54:26 dist-test-slave-r0ck krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 15:54:26.409185 12940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-1/data/instance:
uuid: "f5a89efeefbd4058a871316445d49656"
format_stamp: "Formatted at 2026-09-29 15:54:26 on dist-test-slave-r0ck"
server_key: "575123e07f486c8e85a3b4db48c12088"
server_key_iv: "0a2dd90b93a362f1605e4e985e0bb88f"
server_key_version: "kuduclusterkey@0"
I20260929 15:54:26.410080 12940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f5a89efeefbd4058a871316445d49656"
format_stamp: "Formatted at 2026-09-29 15:54:26 on dist-test-slave-r0ck"
server_key: "575123e07f486c8e85a3b4db48c12088"
server_key_iv: "0a2dd90b93a362f1605e4e985e0bb88f"
server_key_version: "kuduclusterkey@0"
I20260929 15:54:26.415558 12940 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 15:54:26.431442 12964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:26.433049 12940 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 15:54:26.433240 12940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-1/wal
uuid: "f5a89efeefbd4058a871316445d49656"
format_stamp: "Formatted at 2026-09-29 15:54:26 on dist-test-slave-r0ck"
server_key: "575123e07f486c8e85a3b4db48c12088"
server_key_iv: "0a2dd90b93a362f1605e4e985e0bb88f"
server_key_version: "kuduclusterkey@0"
I20260929 15:54:26.433513 12940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:54:26.462316 12940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:54:26.467303 12940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:54:26.467617 12940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:54:26.468643 12940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:54:26.470109 12940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:54:26.470214 12940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:26.470364 12940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:54:26.470450 12940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:26.507696 12940 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38219
I20260929 15:54:26.507786 13076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38219 every 8 connection(s)
I20260929 15:54:26.508965 12940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:54:26.518970 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 12940
I20260929 15:54:26.519141 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:54:26.519721 388 external_mini_cluster.cc:1428] Setting key 7d7b09ca556246a4af899ef162eb0aa2
Sep 29 15:54:26 dist-test-slave-r0ck krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:54:26.532613 13079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:26.512054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36421 (local address 127.0.97.62:42957)
0929 15:54:26.512564 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:26.512570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:26.513644 (+ 1074us) server_negotiation.cc:407] Connection header received
0929 15:54:26.516127 (+ 2483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:26.516131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:26.516212 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:26.516329 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:26.518117 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:26.518903 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:26.520838 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:26.521321 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:26.521510 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:26.525821 (+ 4311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:26.525828 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:26.525833 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:26.525879 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:26.529437 (+ 3558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:26.530378 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:26.530383 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:26.530389 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:26.530518 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:26.530983 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:26.530987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:26.530989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:26.531235 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:26.531385 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:26.532007 (+ 622us) server_negotiation.cc:299] Negotiation successful
0929 15:54:26.532228 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":376,"thread_start_us":249,"threads_started":1}
I20260929 15:54:26.533615 13080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:26.512574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42957 (local address 127.0.97.2:36421)
0929 15:54:26.513472 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:26.513522 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 15:54:26.515830 (+ 2308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:54:26.516581 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:54:26.516592 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:54:26.517132 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:54:26.517928 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:26.517951 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:26.519173 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:26.519199 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:26.520679 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:26.520700 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:26.521599 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:26.521607 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:26.521827 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:26.522611 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:54:26.522644 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:54:26.525578 (+ 2934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:54:26.529594 (+ 4016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:26.529605 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:26.529623 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:26.530227 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:26.530626 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:26.530632 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:26.530635 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:26.530830 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:26.531355 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:54:26.531363 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:54:26.531665 (+ 302us) client_negotiation.cc:769] Sending connection context
0929 15:54:26.531907 (+ 242us) client_negotiation.cc:240] Negotiation successful
0929 15:54:26.532397 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":384,"threads_started":1}
I20260929 15:54:26.535112 13077 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42957
I20260929 15:54:26.535414 13077 heartbeater.cc:464] Registering TS with master...
I20260929 15:54:26.536088 13077 heartbeater.cc:511] Master 127.0.97.62:42957 requested a full tablet report, sending...
I20260929 15:54:26.537569 12721 ts_manager.cc:194] Registered new tserver with Master: f5a89efeefbd4058a871316445d49656 (127.0.97.2:38219)
I20260929 15:54:26.538517 12721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:36421
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 15:54:26.578186 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:40351/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42957
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:54:26.833284 12934 heartbeater.cc:503] Master 127.0.97.62:42957 was elected leader, sending a full tablet report...
W20260929 15:54:26.911067 13083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:54:26.911350 13083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:54:26.911441 13083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:54:26.919093 13083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:54:26.919209 13083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:54:26.919320 13083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:54:26.929100 13083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:40351/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:54:26.930461 13083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:54:26.931728 13083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:54:26.945766 13088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:26.945866 13091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:26.946308 13089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:54:26.951086 13083 server_base.cc:1034] running on GCE node
I20260929 15:54:26.951685 13083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:54:26.952715 13083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:54:26.953989 13083 hybrid_clock.cc:648] HybridClock initialized: now 1790697266953915 us; error 77 us; skew 500 ppm
I20260929 15:54:26.954325 13083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:54:26 dist-test-slave-r0ck krb5kdc[12245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:54:26.958012 13083 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:54:26.959461 13083 webserver.cc:460] Webserver started at http://127.0.97.3:38485/ using document root <none> and password file <none>
I20260929 15:54:26.960139 13083 fs_manager.cc:359] Metadata directory not provided
I20260929 15:54:26.960284 13083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:54:26.960580 13083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 15:54:26 dist-test-slave-r0ck krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 15:54:26.975948 13083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-2/data/instance:
uuid: "9bfd698f9ed44fbe9054024f5efd7ed5"
format_stamp: "Formatted at 2026-09-29 15:54:26 on dist-test-slave-r0ck"
server_key: "d924900406eddd03297b960adcbe64b2"
server_key_iv: "ad0a0c5b9b0f5b84f2c4487bacea56b4"
server_key_version: "kuduclusterkey@0"
I20260929 15:54:26.976738 13083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "9bfd698f9ed44fbe9054024f5efd7ed5"
format_stamp: "Formatted at 2026-09-29 15:54:26 on dist-test-slave-r0ck"
server_key: "d924900406eddd03297b960adcbe64b2"
server_key_iv: "ad0a0c5b9b0f5b84f2c4487bacea56b4"
server_key_version: "kuduclusterkey@0"
I20260929 15:54:26.982272 13083 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 15:54:26.999024 13098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:27.000420 13083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:54:27.000591 13083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-2/wal
uuid: "9bfd698f9ed44fbe9054024f5efd7ed5"
format_stamp: "Formatted at 2026-09-29 15:54:26 on dist-test-slave-r0ck"
server_key: "d924900406eddd03297b960adcbe64b2"
server_key_iv: "ad0a0c5b9b0f5b84f2c4487bacea56b4"
server_key_version: "kuduclusterkey@0"
I20260929 15:54:27.000819 13083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:54:27.019135 13083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:54:27.023258 13083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:54:27.023556 13083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:54:27.024466 13083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:54:27.025910 13083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:54:27.026011 13083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:27.026153 13083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:54:27.026228 13083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:27.058944 13083 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45497
I20260929 15:54:27.058991 13210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45497 every 8 connection(s)
I20260929 15:54:27.060245 13083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:54:27.062701 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 13083
I20260929 15:54:27.062875 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:54:27.063395 388 external_mini_cluster.cc:1428] Setting key f30eba2e2cc7f7290351bc20f6944e98
Sep 29 15:54:27 dist-test-slave-r0ck krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:54:27.080691 13213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:27.062870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52713 (local address 127.0.97.62:42957)
0929 15:54:27.063357 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:27.063361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:27.064518 (+ 1157us) server_negotiation.cc:407] Connection header received
0929 15:54:27.066857 (+ 2339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:27.066861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:27.066925 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:27.067013 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:27.068435 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:27.069188 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:27.070707 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:27.071111 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:27.071204 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:27.074346 (+ 3142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:27.074351 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:27.074354 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:27.074384 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:27.077170 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:27.078115 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:27.078119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:27.078121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:27.078249 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:27.078740 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:27.078745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:27.078749 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:27.079044 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:27.079161 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:27.080231 (+ 1070us) server_negotiation.cc:299] Negotiation successful
0929 15:54:27.080409 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":249,"threads_started":1}
I20260929 15:54:27.081593 13214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:27.063458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42957 (local address 127.0.97.3:52713)
0929 15:54:27.064355 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:27.064396 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 15:54:27.066595 (+ 2199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:54:27.067226 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:54:27.067234 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:54:27.067736 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:54:27.068297 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:27.068313 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:27.069318 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:27.069321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:27.070579 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:27.070592 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:27.071284 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:27.071289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:27.071473 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:27.072099 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:54:27.072121 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:54:27.074158 (+ 2037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:54:27.077310 (+ 3152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:27.077318 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:27.077360 (+ 42us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:27.077934 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:27.078372 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:27.078376 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:27.078379 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:27.078616 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:27.079257 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:54:27.079268 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:54:27.079816 (+ 548us) client_negotiation.cc:769] Sending connection context
0929 15:54:27.080091 (+ 275us) client_negotiation.cc:240] Negotiation successful
0929 15:54:27.080459 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":390,"threads_started":1}
I20260929 15:54:27.082942 13211 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42957
I20260929 15:54:27.083285 13211 heartbeater.cc:464] Registering TS with master...
I20260929 15:54:27.083986 13211 heartbeater.cc:511] Master 127.0.97.62:42957 requested a full tablet report, sending...
I20260929 15:54:27.085515 12721 ts_manager.cc:194] Registered new tserver with Master: 9bfd698f9ed44fbe9054024f5efd7ed5 (127.0.97.3:45497)
I20260929 15:54:27.086316 12721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52713
I20260929 15:54:27.091135 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:54:27.092806 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 12801
Sep 29 15:54:27 dist-test-slave-r0ck krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 15:54:27.109133 13216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:27.093715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41506 (local address 127.0.97.1:38189)
0929 15:54:27.094321 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:27.094327 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:27.094352 (+ 25us) server_negotiation.cc:407] Connection header received
0929 15:54:27.094405 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:27.094411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:27.094587 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:27.094722 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:27.096222 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:27.097238 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:27.098683 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:27.099230 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:27.099340 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:27.103652 (+ 4312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:27.103681 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:27.103694 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:27.103727 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:27.106741 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:27.107373 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:27.107379 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:27.107381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:27.107515 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:27.107825 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:27.107828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:27.107829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:27.108130 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:27.108357 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:27.108669 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 15:54:27.108859 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":308,"threads_started":1}
I20260929 15:54:27.110319 12929 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:54:27.195585 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 12801
I20260929 15:54:27.211539 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 12940
Sep 29 15:54:27 dist-test-slave-r0ck krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 15:54:27.229530 13218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:27.212462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58364 (local address 127.0.97.2:38219)
0929 15:54:27.213019 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:27.213026 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:27.213051 (+ 25us) server_negotiation.cc:407] Connection header received
0929 15:54:27.213110 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:27.213116 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:27.213328 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:27.213539 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:27.214877 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:27.215866 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:27.217198 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:27.217886 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:27.217999 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:27.223391 (+ 5392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:27.223402 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:27.223416 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:27.223459 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:27.226674 (+ 3215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:27.227395 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:27.227402 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:27.227404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:27.227528 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:27.227921 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:27.227924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:27.227926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:27.228266 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:27.228512 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:27.228832 (+ 320us) server_negotiation.cc:299] Negotiation successful
0929 15:54:27.229040 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":300,"threads_started":1}
I20260929 15:54:27.230724 13072 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:54:27.313176 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 12940
I20260929 15:54:27.328547 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 13083
Sep 29 15:54:27 dist-test-slave-r0ck krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 15:54:27.344573 13220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:27.329479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49930 (local address 127.0.97.3:45497)
0929 15:54:27.330035 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:27.330042 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:27.330070 (+ 28us) server_negotiation.cc:407] Connection header received
0929 15:54:27.330133 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:27.330139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:27.330342 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:27.330473 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:27.331731 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:27.332738 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:27.334168 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:27.334727 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:27.334831 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:27.339097 (+ 4266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:27.339111 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:27.339129 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:27.339165 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:27.341921 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:27.342590 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:27.342597 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:27.342599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:27.342724 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:27.343080 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:27.343085 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:27.343087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:27.343401 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:27.343720 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:27.343968 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 15:54:27.344219 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":286,"threads_started":1}
I20260929 15:54:27.345965 13206 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:54:27.428984 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 13083
I20260929 15:54:27.444409 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 12689
I20260929 15:54:27.445684 12751 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:54:27.553036 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 12689
I20260929 15:54:27.573622 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42957
--webserver_interface=127.0.97.62
--webserver_port=45235
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42957
--ranger_config_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:40351/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:40351/kms with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:54:27.890689 13223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:54:27.890964 13223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:54:27.891052 13223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:54:27.898066 13223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:54:27.898175 13223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:54:27.898219 13223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:54:27.898276 13223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:54:27.898324 13223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:54:27.908035 13223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:40351/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42957
--ranger_config_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=45235
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:54:27.909212 13223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:54:27.910460 13223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:54:27.921895 13228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:27.921833 13229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:27.921784 13231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:54:27.922989 13223 server_base.cc:1034] running on GCE node
I20260929 15:54:27.923821 13223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:54:27.925271 13223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:54:27.926478 13223 hybrid_clock.cc:648] HybridClock initialized: now 1790697267926413 us; error 53 us; skew 500 ppm
I20260929 15:54:27.926892 13223 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:54:27 dist-test-slave-r0ck krb5kdc[12245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:54:27.930233 13223 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:54:27.931679 13223 webserver.cc:460] Webserver started at http://127.0.97.62:45235/ using document root <none> and password file <none>
I20260929 15:54:27.932307 13223 fs_manager.cc:359] Metadata directory not provided
I20260929 15:54:27.932416 13223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 15:54:27 dist-test-slave-r0ck krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 15:54:27.981216 13223 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 15:54:27.984120 13239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:27.985599 13223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:54:27.985787 13223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "576b6935026a416e9d336bd069ec551e"
format_stamp: "Formatted at 2026-09-29 15:54:24 on dist-test-slave-r0ck"
server_key: "681f188b63102d291ec61903d2243310"
server_key_iv: "f9572bcbedef15a3577fc903746c4c38"
server_key_version: "kuduclusterkey@0"
I20260929 15:54:27.986832 13223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:54:28.010190 13223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:54:28.016167 13223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:54:28.016475 13223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:54:28.034142 13223 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42957
I20260929 15:54:28.034150 13290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42957 every 8 connection(s)
I20260929 15:54:28.035673 13223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:54:28.042174 13291 sys_catalog.cc:263] Verifying existing consensus state
I20260929 15:54:28.045226 13291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e: Bootstrap starting.
I20260929 15:54:28.045318 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 13223
I20260929 15:54:28.046150 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:40351/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:38189
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:42957
--webserver_port=46653
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:40351/kms with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:54:28.068897 13291 log.cc:824] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e: Log is configured to *not* fsync() on all Append() calls
I20260929 15:54:28.079965 13291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 15:54:28.080600 13291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e: Bootstrap complete.
I20260929 15:54:28.093602 13291 raft_consensus.cc:348] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "576b6935026a416e9d336bd069ec551e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42957 } }
I20260929 15:54:28.094758 13291 raft_consensus.cc:729] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 576b6935026a416e9d336bd069ec551e, State: Initialized, Role: FOLLOWER
I20260929 15:54:28.095436 13291 consensus_queue.cc:260] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "576b6935026a416e9d336bd069ec551e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42957 } }
I20260929 15:54:28.095685 13291 raft_consensus.cc:388] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:54:28.095806 13291 raft_consensus.cc:482] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:54:28.095961 13291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e [term 1 FOLLOWER]: Advancing to term 2
I20260929 15:54:28.101447 13291 raft_consensus.cc:504] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "576b6935026a416e9d336bd069ec551e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42957 } }
I20260929 15:54:28.102007 13291 leader_election.cc:302] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 576b6935026a416e9d336bd069ec551e; no voters:
I20260929 15:54:28.102661 13291 leader_election.cc:288] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 15:54:28.102823 13295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e [term 2 FOLLOWER]: Leader election won for term 2
I20260929 15:54:28.104038 13295 raft_consensus.cc:686] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e [term 2 LEADER]: Becoming Leader. State: Replica: 576b6935026a416e9d336bd069ec551e, State: Running, Role: LEADER
I20260929 15:54:28.104578 13295 consensus_queue.cc:237] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "576b6935026a416e9d336bd069ec551e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42957 } }
I20260929 15:54:28.104856 13291 sys_catalog.cc:564] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:54:28.106954 13296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "576b6935026a416e9d336bd069ec551e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "576b6935026a416e9d336bd069ec551e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42957 } } }
I20260929 15:54:28.107417 13296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e [sys.catalog]: This master's current role is: LEADER
I20260929 15:54:28.108191 13297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 576b6935026a416e9d336bd069ec551e. Latest consensus state: current_term: 2 leader_uuid: "576b6935026a416e9d336bd069ec551e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "576b6935026a416e9d336bd069ec551e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42957 } } }
I20260929 15:54:28.110069 13297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e [sys.catalog]: This master's current role is: LEADER
I20260929 15:54:28.116557 13302 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:54:28.122262 13302 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:54:28.123677 13302 catalog_manager.cc:1259] Loaded cluster ID: 67d95d0d91604039b55cb632a146d2eb
I20260929 15:54:28.123775 13302 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:54:28.127597 13302 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:54:28.128983 13302 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e: Loaded TSK: 0
I20260929 15:54:28.130510 13302 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:54:28.134022 13291 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260929 15:54:28.503686 13293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:54:28.504079 13293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:54:28.504278 13293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:54:28.515893 13293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:54:28.516110 13293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:54:28.516348 13293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:54:28.532052 13293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:40351/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:38189
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=46653
--tserver_master_addrs=127.0.97.62:42957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:54:28.534015 13293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:54:28.535763 13293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:54:28.565819 13330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:28.566190 13329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:54:28.568585 13293 server_base.cc:1034] running on GCE node
W20260929 15:54:28.568861 13332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:54:28.569769 13293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:54:28.571053 13293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:54:28.572341 13293 hybrid_clock.cc:648] HybridClock initialized: now 1790697268572277 us; error 51 us; skew 500 ppm
I20260929 15:54:28.572743 13293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:54:28 dist-test-slave-r0ck krb5kdc[12245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:54:28.577004 13293 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:54:28.590365 13293 webserver.cc:460] Webserver started at http://127.0.97.1:46653/ using document root <none> and password file <none>
I20260929 15:54:28.591123 13293 fs_manager.cc:359] Metadata directory not provided
I20260929 15:54:28.591280 13293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 15:54:28 dist-test-slave-r0ck krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 15:54:28.625770 13293 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.006s sys 0.007s
I20260929 15:54:28.630703 13340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:28.632453 13293 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260929 15:54:28.632728 13293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-0/wal
uuid: "fe46a8f385dd458f8992090bf3a66bb7"
format_stamp: "Formatted at 2026-09-29 15:54:25 on dist-test-slave-r0ck"
server_key: "f63cc0bdbd40eb9f46ba78358ae17209"
server_key_iv: "1fbaa2faa397f63375df7c04963d95d3"
server_key_version: "kuduclusterkey@0"
I20260929 15:54:28.634253 13293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:54:28.663718 13293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:54:28.668879 13293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:54:28.669250 13293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:54:28.673750 13293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:54:28.681808 13293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:54:28.681986 13293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260929 15:54:28.682193 13293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:54:28.682302 13293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:28.722864 13293 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38189
I20260929 15:54:28.722935 13452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38189 every 8 connection(s)
I20260929 15:54:28.724869 13293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:54:28.726459 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 13293
I20260929 15:54:28.727293 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:40351/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:38219
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:42957
--webserver_port=45595
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:40351/kms with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 15:54:28 dist-test-slave-r0ck krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:54:28.777762 13458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:28.735652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42957 (local address 127.0.97.1:53867)
0929 15:54:28.748503 (+ 12851us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:28.748567 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 15:54:28.751208 (+ 2641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:54:28.755697 (+ 4489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:54:28.755719 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:54:28.756342 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:54:28.757105 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:28.757132 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:28.759569 (+ 2437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:28.759578 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:28.762219 (+ 2641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:28.762250 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:28.764023 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:28.764032 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:28.764325 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:28.765280 (+ 955us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:54:28.765311 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:54:28.768254 (+ 2943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:54:28.772642 (+ 4388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:28.772650 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:28.772666 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:28.773242 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:28.773790 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:28.773795 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:28.773799 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:28.774054 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:28.775105 (+ 1051us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:54:28.775113 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:54:28.775468 (+ 355us) client_negotiation.cc:769] Sending connection context
0929 15:54:28.775746 (+ 278us) client_negotiation.cc:240] Negotiation successful
0929 15:54:28.776293 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12507,"thread_start_us":15969,"threads_started":1}
I20260929 15:54:28.780244 13457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:28.735527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53867 (local address 127.0.97.62:42957)
0929 15:54:28.745373 (+ 9846us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:28.745396 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:28.753395 (+ 7999us) server_negotiation.cc:407] Connection header received
0929 15:54:28.753893 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:28.753915 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:28.754420 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:28.754836 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:28.757401 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:28.759370 (+ 1969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:28.762461 (+ 3091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:28.763759 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:28.764006 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:28.768573 (+ 4567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:28.768584 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:28.768604 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:28.768654 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:28.772472 (+ 3818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:28.773435 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:28.773445 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:28.773453 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:28.773658 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:28.774182 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:28.774187 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:28.774191 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:28.774832 (+ 641us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:28.775017 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:28.778786 (+ 3769us) server_negotiation.cc:299] Negotiation successful
0929 15:54:28.779166 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9496,"thread_start_us":9747,"threads_started":1}
I20260929 15:54:28.783859 13453 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42957
I20260929 15:54:28.784186 13453 heartbeater.cc:464] Registering TS with master...
I20260929 15:54:28.785010 13453 heartbeater.cc:511] Master 127.0.97.62:42957 requested a full tablet report, sending...
W20260929 15:54:28.788242 13453 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42957 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 15:54:28.788488 13453 heartbeater.cc:464] Registering TS with master...
I20260929 15:54:28.788758 13453 heartbeater.cc:511] Master 127.0.97.62:42957 requested a full tablet report, sending...
I20260929 15:54:28.790081 13453 heartbeater.cc:464] Registering TS with master...
I20260929 15:54:28.790354 13453 heartbeater.cc:511] Master 127.0.97.62:42957 requested a full tablet report, sending...
W20260929 15:54:28.791560 13453 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 15:54:29.129246 13456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:54:29.129628 13456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:54:29.129796 13456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:54:29.137166 13456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:54:29.137300 13456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:54:29.137507 13456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:54:29.147413 13456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:40351/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:38219
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=45595
--tserver_master_addrs=127.0.97.62:42957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:54:29.148710 13456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:54:29.150031 13456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:54:29.164368 13471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:29.164991 13470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:29.165205 13473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:54:29.165822 13456 server_base.cc:1034] running on GCE node
I20260929 15:54:29.166486 13456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:54:29.167394 13456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:54:29.168651 13456 hybrid_clock.cc:648] HybridClock initialized: now 1790697269168588 us; error 48 us; skew 500 ppm
I20260929 15:54:29.168977 13456 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:54:29 dist-test-slave-r0ck krb5kdc[12245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:54:29.172187 13456 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:54:29.173720 13456 webserver.cc:460] Webserver started at http://127.0.97.2:45595/ using document root <none> and password file <none>
I20260929 15:54:29.174373 13456 fs_manager.cc:359] Metadata directory not provided
I20260929 15:54:29.174508 13456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 15:54:29 dist-test-slave-r0ck krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 15:54:29.197932 13456 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 15:54:29.200701 13481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:29.202147 13456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:54:29.202312 13456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-1/wal
uuid: "f5a89efeefbd4058a871316445d49656"
format_stamp: "Formatted at 2026-09-29 15:54:26 on dist-test-slave-r0ck"
server_key: "575123e07f486c8e85a3b4db48c12088"
server_key_iv: "0a2dd90b93a362f1605e4e985e0bb88f"
server_key_version: "kuduclusterkey@0"
I20260929 15:54:29.203639 13456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:54:29.232164 13456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:54:29.237232 13456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:54:29.237576 13456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:54:29.238711 13456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:54:29.240688 13456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:54:29.240772 13456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:29.240842 13456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:54:29.240888 13456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:29.305166 13456 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38219
I20260929 15:54:29.307112 13456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:54:29.307691 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 13456
I20260929 15:54:29.308534 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:40351/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:45497
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:42957
--webserver_port=38485
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:40351/kms with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:54:29.305186 13595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38219 every 8 connection(s)
Sep 29 15:54:29 dist-test-slave-r0ck krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:54:29.375990 13599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:29.314993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42957 (local address 127.0.97.2:57957)
0929 15:54:29.327660 (+ 12667us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:29.327721 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 15:54:29.341569 (+ 13848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:54:29.346269 (+ 4700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:54:29.346284 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:54:29.346934 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:54:29.347850 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:29.347877 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:29.349508 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:29.349514 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:29.351644 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:29.351659 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:29.362726 (+ 11067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:29.362734 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:29.363080 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:29.363941 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:54:29.363975 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:54:29.366824 (+ 2849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:54:29.371147 (+ 4323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:29.371157 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:29.371174 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:29.371781 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:29.372279 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:29.372286 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:29.372292 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:29.372545 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:29.373237 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:54:29.373245 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:54:29.373687 (+ 442us) client_negotiation.cc:769] Sending connection context
0929 15:54:29.374035 (+ 348us) client_negotiation.cc:240] Negotiation successful
0929 15:54:29.374591 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12348,"thread_start_us":12311,"threads_started":1}
I20260929 15:54:29.377132 13600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:29.315538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57957 (local address 127.0.97.62:42957)
0929 15:54:29.326133 (+ 10595us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:29.326143 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:29.329408 (+ 3265us) server_negotiation.cc:407] Connection header received
0929 15:54:29.345465 (+ 16057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:29.345472 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:29.345577 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:29.345726 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:29.348097 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:29.349265 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:29.361495 (+ 12230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:29.362188 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:29.362312 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:29.367139 (+ 4827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:29.367146 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:29.367152 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:29.367194 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:29.370903 (+ 3709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:29.371936 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:29.371941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:29.371945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:29.372138 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:29.372689 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:29.372694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:29.372697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:29.373012 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:29.373154 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:29.376533 (+ 3379us) server_negotiation.cc:299] Negotiation successful
0929 15:54:29.376754 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10431,"thread_start_us":10325,"threads_started":1}
I20260929 15:54:29.378464 13596 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42957
I20260929 15:54:29.378809 13596 heartbeater.cc:464] Registering TS with master...
I20260929 15:54:29.379658 13596 heartbeater.cc:511] Master 127.0.97.62:42957 requested a full tablet report, sending...
I20260929 15:54:29.381798 13256 ts_manager.cc:194] Registered new tserver with Master: f5a89efeefbd4058a871316445d49656 (127.0.97.2:38219)
I20260929 15:54:29.383951 13256 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57957
Sep 29 15:54:29 dist-test-slave-r0ck krb5kdc[12245](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790697269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 15:54:29.782836 13601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:54:29.783311 13601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:54:29.783543 13601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 15:54:29.791710 13453 heartbeater.cc:464] Registering TS with master...
I20260929 15:54:29.792102 13453 heartbeater.cc:511] Master 127.0.97.62:42957 requested a full tablet report, sending...
I20260929 15:54:29.793383 13256 ts_manager.cc:194] Registered new tserver with Master: fe46a8f385dd458f8992090bf3a66bb7 (127.0.97.1:38189)
I20260929 15:54:29.794425 13256 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53867
W20260929 15:54:29.794945 13601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:54:29.795113 13601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:54:29.795315 13601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:54:29.810644 13601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:40351/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:45497
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=38485
--tserver_master_addrs=127.0.97.62:42957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:54:29.812301 13601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:54:29.813917 13601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:54:29.840325 13609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:29.842926 13608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:29.845660 13612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:54:29.857592 13601 server_base.cc:1034] running on GCE node
I20260929 15:54:29.858934 13601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:54:29.860378 13601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:54:29.861696 13601 hybrid_clock.cc:648] HybridClock initialized: now 1790697269861644 us; error 42 us; skew 500 ppm
I20260929 15:54:29.862046 13601 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:54:29 dist-test-slave-r0ck krb5kdc[12245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:54:29.865928 13601 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
Sep 29 15:54:29 dist-test-slave-r0ck krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 15:54:29.884853 13601 webserver.cc:460] Webserver started at http://127.0.97.3:38485/ using document root <none> and password file <none>
I20260929 15:54:29.885818 13601 fs_manager.cc:359] Metadata directory not provided
I20260929 15:54:29.885972 13601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 15:54:29 dist-test-slave-r0ck krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 15:54:29.923566 13601 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260929 15:54:29.937589 13619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:29.940896 13601 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.012s sys 0.000s
I20260929 15:54:29.941087 13601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-2/wal
uuid: "9bfd698f9ed44fbe9054024f5efd7ed5"
format_stamp: "Formatted at 2026-09-29 15:54:26 on dist-test-slave-r0ck"
server_key: "d924900406eddd03297b960adcbe64b2"
server_key_iv: "ad0a0c5b9b0f5b84f2c4487bacea56b4"
server_key_version: "kuduclusterkey@0"
I20260929 15:54:29.942442 13601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:54:29.970345 13601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:54:29.975057 13601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:54:29.975337 13601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:54:29.976413 13601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:54:29.977988 13601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:54:29.978087 13601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:29.978191 13601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:54:29.978262 13601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:30.026152 13601 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45497
I20260929 15:54:30.026218 13731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45497 every 8 connection(s)
I20260929 15:54:30.027983 13601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:54:30.036089 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 13601
Sep 29 15:54:30 dist-test-slave-r0ck krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:54:30.073900 13736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:30.037102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53162 (local address 127.0.97.62:42957)
0929 15:54:30.051972 (+ 14870us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:30.051977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:30.051999 (+ 22us) server_negotiation.cc:407] Connection header received
0929 15:54:30.052062 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:30.052067 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:30.052167 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:30.052285 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:30.056656 (+ 4371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:30.057970 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:30.059931 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:30.060569 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:30.060689 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:30.064589 (+ 3900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:30.064595 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:30.064600 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:30.064637 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:30.069884 (+ 5247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:30.070800 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:30.070805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:30.070809 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:30.070990 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:30.071442 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:30.071447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:30.071450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:30.071780 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:30.072024 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:30.073261 (+ 1237us) server_negotiation.cc:299] Negotiation successful
0929 15:54:30.073514 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14744,"thread_start_us":15298,"threads_started":1}
I20260929 15:54:30.075670 13734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:30.030938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36945 (local address 127.0.97.62:42957)
0929 15:54:30.051125 (+ 20187us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:30.051135 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:30.052209 (+ 1074us) server_negotiation.cc:407] Connection header received
0929 15:54:30.054387 (+ 2178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:30.054393 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:30.054479 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:30.054585 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:30.056622 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:30.057939 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:30.062223 (+ 4284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:30.062891 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:30.062984 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:30.067365 (+ 4381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:30.067372 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:30.067377 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:30.067444 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:30.071229 (+ 3785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:30.072172 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:30.072177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:30.072180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:30.072329 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:30.073286 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:30.073292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:30.073296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:30.073667 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:30.073803 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:30.075066 (+ 1263us) server_negotiation.cc:299] Negotiation successful
0929 15:54:30.075271 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19992,"thread_start_us":21258,"threads_started":1}
I20260929 15:54:30.077973 13735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:30.031651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42957 (local address 127.0.97.3:36945)
0929 15:54:30.051709 (+ 20058us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:30.051756 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 15:54:30.054084 (+ 2328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:54:30.055029 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:54:30.055042 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:54:30.055644 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:54:30.056423 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:30.056447 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:30.059983 (+ 3536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:30.059989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:30.062050 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:30.062064 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:30.063320 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:30.063328 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:30.063608 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:30.064328 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:54:30.064359 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:54:30.067081 (+ 2722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:54:30.071396 (+ 4315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:30.071405 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:30.071423 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:30.072014 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:30.072479 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:30.072484 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:30.072487 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:30.072765 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:30.073907 (+ 1142us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:54:30.073915 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:54:30.074612 (+ 697us) client_negotiation.cc:769] Sending connection context
0929 15:54:30.076008 (+ 1396us) client_negotiation.cc:240] Negotiation successful
0929 15:54:30.076552 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19756,"thread_start_us":19530,"threads_started":1}
I20260929 15:54:30.079636 13732 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42957
I20260929 15:54:30.079957 13732 heartbeater.cc:464] Registering TS with master...
I20260929 15:54:30.080818 13732 heartbeater.cc:511] Master 127.0.97.62:42957 requested a full tablet report, sending...
I20260929 15:54:30.082576 13256 ts_manager.cc:194] Registered new tserver with Master: 9bfd698f9ed44fbe9054024f5efd7ed5 (127.0.97.3:45497)
I20260929 15:54:30.083715 13256 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:36945
I20260929 15:54:30.088826 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:54:30.114899 13734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:30.092584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53168 (local address 127.0.97.62:42957)
0929 15:54:30.092761 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:30.092767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:30.098247 (+ 5480us) server_negotiation.cc:407] Connection header received
0929 15:54:30.098434 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:30.098440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:30.098526 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:30.098637 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:30.100622 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:30.101745 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:30.103753 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:30.104395 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:30.104519 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:30.108289 (+ 3770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:30.108296 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:30.108302 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:30.108343 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:30.111914 (+ 3571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:30.112766 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:30.112772 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:30.112775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:30.112928 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:30.113365 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:30.113370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:30.113374 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:30.113718 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:30.113966 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:30.114332 (+ 366us) server_negotiation.cc:299] Negotiation successful
0929 15:54:30.114558 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 15:54:30.122149 13256 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53168:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:54:30.133941 13256 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:54:30.182415 13750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:30.151949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38219 (local address 127.0.0.1:50616)
0929 15:54:30.164330 (+ 12381us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:30.164373 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 15:54:30.164556 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:54:30.171850 (+ 7294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:54:30.171859 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:54:30.171914 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:54:30.172380 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:30.172399 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:30.175522 (+ 3123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:30.175528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:30.178036 (+ 2508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:30.178052 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:30.180203 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:30.180210 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:30.180451 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:30.181660 (+ 1209us) client_negotiation.cc:769] Sending connection context
0929 15:54:30.181783 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 15:54:30.181979 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12130,"thread_start_us":20497,"threads_started":1}
I20260929 15:54:30.183995 13748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:30.152193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50616 (local address 127.0.97.2:38219)
0929 15:54:30.170996 (+ 18803us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:30.171010 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:30.171052 (+ 42us) server_negotiation.cc:407] Connection header received
0929 15:54:30.171183 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:30.171194 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:30.171537 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:30.171818 (+ 281us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:54:30.172644 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:30.175352 (+ 2708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:30.178262 (+ 2910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:30.179901 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:30.183215 (+ 3314us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:30.183420 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:30.183508 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 15:54:30.183676 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18568,"thread_start_us":19865,"threads_started":1}
I20260929 15:54:30.187402 13747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:30.152591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45497 (local address 127.0.0.1:49938)
0929 15:54:30.172723 (+ 20132us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:30.172745 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:54:30.172966 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:54:30.174047 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:54:30.174054 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:54:30.174088 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:54:30.174528 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:30.174542 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:30.177187 (+ 2645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:30.177195 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:30.179923 (+ 2728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:30.179943 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:30.186588 (+ 6645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:30.186594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:30.186770 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:30.186804 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 15:54:30.186897 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 15:54:30.187061 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20001,"thread_start_us":11351,"threads_started":1}
I20260929 15:54:30.188215 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:30.152324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49938 (local address 127.0.97.3:45497)
0929 15:54:30.162342 (+ 10018us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:30.162355 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:30.172853 (+ 10498us) server_negotiation.cc:407] Connection header received
0929 15:54:30.173165 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:30.173172 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:30.173428 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:30.173626 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:54:30.174788 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:30.176947 (+ 2159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:30.184502 (+ 7555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:30.186088 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:30.186230 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:30.186445 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:30.187692 (+ 1247us) server_negotiation.cc:299] Negotiation successful
0929 15:54:30.187858 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9811,"thread_start_us":9665,"threads_started":1}
I20260929 15:54:30.193579 13667 tablet_service.cc:1467] Processing CreateTablet for tablet aeab19d5e3e345d7a2f34a2ca1749b2b (DEFAULT_TABLE table=test-table [id=51e25a94f97046628ad5ff29d0038a92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:54:30.195456 13667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeab19d5e3e345d7a2f34a2ca1749b2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:54:30.209630 13750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:30.179137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38189 (local address 127.0.0.1:38610)
0929 15:54:30.182926 (+ 3789us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:30.182945 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:54:30.183045 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:54:30.202134 (+ 19089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:54:30.202138 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:54:30.202162 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:54:30.202301 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:30.202308 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:30.204756 (+ 2448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:30.204764 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:30.206836 (+ 2072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:30.206853 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:30.208892 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:30.208898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:30.209077 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:30.209113 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 15:54:30.209193 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 15:54:30.209332 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3686,"thread_start_us":25768,"threads_started":1}
I20260929 15:54:30.210067 13531 tablet_service.cc:1467] Processing CreateTablet for tablet aeab19d5e3e345d7a2f34a2ca1749b2b (DEFAULT_TABLE table=test-table [id=51e25a94f97046628ad5ff29d0038a92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:54:30.210995 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:30.173158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38610 (local address 127.0.97.1:38189)
0929 15:54:30.189858 (+ 16700us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:30.189870 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:30.189908 (+ 38us) server_negotiation.cc:407] Connection header received
0929 15:54:30.201565 (+ 11657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:30.201577 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:30.201892 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:30.202149 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:54:30.202445 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:30.204564 (+ 2119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:30.207042 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:30.208671 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:30.210144 (+ 1473us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:30.210362 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:30.210442 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 15:54:30.210594 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16517,"thread_start_us":16349,"threads_started":1}
I20260929 15:54:30.211402 13531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeab19d5e3e345d7a2f34a2ca1749b2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:54:30.214524 13388 tablet_service.cc:1467] Processing CreateTablet for tablet aeab19d5e3e345d7a2f34a2ca1749b2b (DEFAULT_TABLE table=test-table [id=51e25a94f97046628ad5ff29d0038a92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:54:30.215902 13388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeab19d5e3e345d7a2f34a2ca1749b2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:54:30.232899 13755 tablet_bootstrap.cc:492] T aeab19d5e3e345d7a2f34a2ca1749b2b P f5a89efeefbd4058a871316445d49656: Bootstrap starting.
I20260929 15:54:30.238487 13755 tablet_bootstrap.cc:654] T aeab19d5e3e345d7a2f34a2ca1749b2b P f5a89efeefbd4058a871316445d49656: Neither blocks nor log segments found. Creating new log.
I20260929 15:54:30.239392 13754 tablet_bootstrap.cc:492] T aeab19d5e3e345d7a2f34a2ca1749b2b P 9bfd698f9ed44fbe9054024f5efd7ed5: Bootstrap starting.
I20260929 15:54:30.239749 13755 log.cc:824] T aeab19d5e3e345d7a2f34a2ca1749b2b P f5a89efeefbd4058a871316445d49656: Log is configured to *not* fsync() on all Append() calls
I20260929 15:54:30.244122 13754 tablet_bootstrap.cc:654] T aeab19d5e3e345d7a2f34a2ca1749b2b P 9bfd698f9ed44fbe9054024f5efd7ed5: Neither blocks nor log segments found. Creating new log.
I20260929 15:54:30.246004 13754 log.cc:824] T aeab19d5e3e345d7a2f34a2ca1749b2b P 9bfd698f9ed44fbe9054024f5efd7ed5: Log is configured to *not* fsync() on all Append() calls
I20260929 15:54:30.246292 13755 tablet_bootstrap.cc:492] T aeab19d5e3e345d7a2f34a2ca1749b2b P f5a89efeefbd4058a871316445d49656: No bootstrap required, opened a new log
I20260929 15:54:30.246601 13755 ts_tablet_manager.cc:1397] T aeab19d5e3e345d7a2f34a2ca1749b2b P f5a89efeefbd4058a871316445d49656: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.006s
I20260929 15:54:30.254173 13756 tablet_bootstrap.cc:492] T aeab19d5e3e345d7a2f34a2ca1749b2b P fe46a8f385dd458f8992090bf3a66bb7: Bootstrap starting.
I20260929 15:54:30.258225 13756 tablet_bootstrap.cc:654] T aeab19d5e3e345d7a2f34a2ca1749b2b P fe46a8f385dd458f8992090bf3a66bb7: Neither blocks nor log segments found. Creating new log.
I20260929 15:54:30.258417 13754 tablet_bootstrap.cc:492] T aeab19d5e3e345d7a2f34a2ca1749b2b P 9bfd698f9ed44fbe9054024f5efd7ed5: No bootstrap required, opened a new log
I20260929 15:54:30.258752 13754 ts_tablet_manager.cc:1397] T aeab19d5e3e345d7a2f34a2ca1749b2b P 9bfd698f9ed44fbe9054024f5efd7ed5: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.004s
Sep 29 15:54:30 dist-test-slave-r0ck krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 15:54:30.259552 13756 log.cc:824] T aeab19d5e3e345d7a2f34a2ca1749b2b P fe46a8f385dd458f8992090bf3a66bb7: Log is configured to *not* fsync() on all Append() calls
I20260929 15:54:30.264256 13755 raft_consensus.cc:348] T aeab19d5e3e345d7a2f34a2ca1749b2b P f5a89efeefbd4058a871316445d49656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfd698f9ed44fbe9054024f5efd7ed5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45497 } } peers { permanent_uuid: "f5a89efeefbd4058a871316445d49656" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38219 } } peers { permanent_uuid: "fe46a8f385dd458f8992090bf3a66bb7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38189 } }
I20260929 15:54:30.264739 13755 raft_consensus.cc:374] T aeab19d5e3e345d7a2f34a2ca1749b2b P f5a89efeefbd4058a871316445d49656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:54:30.264838 13755 raft_consensus.cc:729] T aeab19d5e3e345d7a2f34a2ca1749b2b P f5a89efeefbd4058a871316445d49656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5a89efeefbd4058a871316445d49656, State: Initialized, Role: FOLLOWER
I20260929 15:54:30.265590 13755 consensus_queue.cc:260] T aeab19d5e3e345d7a2f34a2ca1749b2b P f5a89efeefbd4058a871316445d49656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfd698f9ed44fbe9054024f5efd7ed5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45497 } } peers { permanent_uuid: "f5a89efeefbd4058a871316445d49656" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38219 } } peers { permanent_uuid: "fe46a8f385dd458f8992090bf3a66bb7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38189 } }
I20260929 15:54:30.269260 13754 raft_consensus.cc:348] T aeab19d5e3e345d7a2f34a2ca1749b2b P 9bfd698f9ed44fbe9054024f5efd7ed5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfd698f9ed44fbe9054024f5efd7ed5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45497 } } peers { permanent_uuid: "f5a89efeefbd4058a871316445d49656" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38219 } } peers { permanent_uuid: "fe46a8f385dd458f8992090bf3a66bb7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38189 } }
I20260929 15:54:30.269718 13754 raft_consensus.cc:374] T aeab19d5e3e345d7a2f34a2ca1749b2b P 9bfd698f9ed44fbe9054024f5efd7ed5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:54:30.269819 13754 raft_consensus.cc:729] T aeab19d5e3e345d7a2f34a2ca1749b2b P 9bfd698f9ed44fbe9054024f5efd7ed5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfd698f9ed44fbe9054024f5efd7ed5, State: Initialized, Role: FOLLOWER
I20260929 15:54:30.270529 13754 consensus_queue.cc:260] T aeab19d5e3e345d7a2f34a2ca1749b2b P 9bfd698f9ed44fbe9054024f5efd7ed5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfd698f9ed44fbe9054024f5efd7ed5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45497 } } peers { permanent_uuid: "f5a89efeefbd4058a871316445d49656" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38219 } } peers { permanent_uuid: "fe46a8f385dd458f8992090bf3a66bb7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38189 } }
I20260929 15:54:30.277895 13756 tablet_bootstrap.cc:492] T aeab19d5e3e345d7a2f34a2ca1749b2b P fe46a8f385dd458f8992090bf3a66bb7: No bootstrap required, opened a new log
I20260929 15:54:30.278270 13756 ts_tablet_manager.cc:1397] T aeab19d5e3e345d7a2f34a2ca1749b2b P fe46a8f385dd458f8992090bf3a66bb7: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.000s
I20260929 15:54:30.282343 13596 heartbeater.cc:503] Master 127.0.97.62:42957 was elected leader, sending a full tablet report...
I20260929 15:54:30.283479 13755 ts_tablet_manager.cc:1428] T aeab19d5e3e345d7a2f34a2ca1749b2b P f5a89efeefbd4058a871316445d49656: Time spent starting tablet: real 0.037s user 0.012s sys 0.007s
I20260929 15:54:30.291875 13756 raft_consensus.cc:348] T aeab19d5e3e345d7a2f34a2ca1749b2b P fe46a8f385dd458f8992090bf3a66bb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfd698f9ed44fbe9054024f5efd7ed5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45497 } } peers { permanent_uuid: "f5a89efeefbd4058a871316445d49656" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38219 } } peers { permanent_uuid: "fe46a8f385dd458f8992090bf3a66bb7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38189 } }
I20260929 15:54:30.292361 13756 raft_consensus.cc:374] T aeab19d5e3e345d7a2f34a2ca1749b2b P fe46a8f385dd458f8992090bf3a66bb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:54:30.292469 13756 raft_consensus.cc:729] T aeab19d5e3e345d7a2f34a2ca1749b2b P fe46a8f385dd458f8992090bf3a66bb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe46a8f385dd458f8992090bf3a66bb7, State: Initialized, Role: FOLLOWER
I20260929 15:54:30.293241 13756 consensus_queue.cc:260] T aeab19d5e3e345d7a2f34a2ca1749b2b P fe46a8f385dd458f8992090bf3a66bb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfd698f9ed44fbe9054024f5efd7ed5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45497 } } peers { permanent_uuid: "f5a89efeefbd4058a871316445d49656" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38219 } } peers { permanent_uuid: "fe46a8f385dd458f8992090bf3a66bb7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38189 } }
I20260929 15:54:30.301512 13732 heartbeater.cc:503] Master 127.0.97.62:42957 was elected leader, sending a full tablet report...
I20260929 15:54:30.301977 13754 ts_tablet_manager.cc:1428] T aeab19d5e3e345d7a2f34a2ca1749b2b P 9bfd698f9ed44fbe9054024f5efd7ed5: Time spent starting tablet: real 0.043s user 0.031s sys 0.008s
I20260929 15:54:30.304405 13453 heartbeater.cc:503] Master 127.0.97.62:42957 was elected leader, sending a full tablet report...
I20260929 15:54:30.305094 13756 ts_tablet_manager.cc:1428] T aeab19d5e3e345d7a2f34a2ca1749b2b P fe46a8f385dd458f8992090bf3a66bb7: Time spent starting tablet: real 0.027s user 0.017s sys 0.008s
W20260929 15:54:30.355007 13598 tablet.cc:2369] T aeab19d5e3e345d7a2f34a2ca1749b2b P f5a89efeefbd4058a871316445d49656: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:54:30.484360 13454 tablet.cc:2369] T aeab19d5e3e345d7a2f34a2ca1749b2b P fe46a8f385dd458f8992090bf3a66bb7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:54:30.547721 13733 tablet.cc:2369] T aeab19d5e3e345d7a2f34a2ca1749b2b P 9bfd698f9ed44fbe9054024f5efd7ed5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:54:30.571215 13762 raft_consensus.cc:482] T aeab19d5e3e345d7a2f34a2ca1749b2b P fe46a8f385dd458f8992090bf3a66bb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:54:30.571545 13762 raft_consensus.cc:504] T aeab19d5e3e345d7a2f34a2ca1749b2b P fe46a8f385dd458f8992090bf3a66bb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfd698f9ed44fbe9054024f5efd7ed5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45497 } } peers { permanent_uuid: "f5a89efeefbd4058a871316445d49656" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38219 } } peers { permanent_uuid: "fe46a8f385dd458f8992090bf3a66bb7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38189 } }
I20260929 15:54:30.573679 13762 leader_election.cc:288] T aeab19d5e3e345d7a2f34a2ca1749b2b P fe46a8f385dd458f8992090bf3a66bb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bfd698f9ed44fbe9054024f5efd7ed5 (127.0.97.3:45497), f5a89efeefbd4058a871316445d49656 (127.0.97.2:38219)
I20260929 15:54:30.583652 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:30.574350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55389 (local address 127.0.97.3:45497)
0929 15:54:30.574560 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:30.574566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:30.575056 (+ 490us) server_negotiation.cc:407] Connection header received
0929 15:54:30.575445 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:30.575454 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:30.575562 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:30.575649 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:54:30.576661 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:30.578098 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:30.581409 (+ 3311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:30.582487 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:30.582632 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:30.582740 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:30.583080 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 15:54:30.583215 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 15:54:30.583823 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:30.574102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45497 (local address 127.0.97.1:55389)
0929 15:54:30.574936 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:30.574962 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 15:54:30.575198 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:54:30.575909 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:54:30.575917 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:54:30.575954 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:54:30.576459 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:30.576475 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:30.578342 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:30.578350 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:30.581154 (+ 2804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:30.581184 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:30.582699 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:30.582705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:30.582927 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:30.582951 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 15:54:30.583055 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 15:54:30.583182 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":497,"threads_started":1}
I20260929 15:54:30.584820 13687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeab19d5e3e345d7a2f34a2ca1749b2b" candidate_uuid: "fe46a8f385dd458f8992090bf3a66bb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfd698f9ed44fbe9054024f5efd7ed5" is_pre_election: true
I20260929 15:54:30.585299 13687 raft_consensus.cc:2454] T aeab19d5e3e345d7a2f34a2ca1749b2b P 9bfd698f9ed44fbe9054024f5efd7ed5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe46a8f385dd458f8992090bf3a66bb7 in term 0.
I20260929 15:54:30.586025 13341 leader_election.cc:302] T aeab19d5e3e345d7a2f34a2ca1749b2b P fe46a8f385dd458f8992090bf3a66bb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfd698f9ed44fbe9054024f5efd7ed5, fe46a8f385dd458f8992090bf3a66bb7; no voters:
I20260929 15:54:30.586452 13762 raft_consensus.cc:2781] T aeab19d5e3e345d7a2f34a2ca1749b2b P fe46a8f385dd458f8992090bf3a66bb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:54:30.586658 13762 raft_consensus.cc:482] T aeab19d5e3e345d7a2f34a2ca1749b2b P fe46a8f385dd458f8992090bf3a66bb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:54:30.586782 13762 raft_consensus.cc:3037] T aeab19d5e3e345d7a2f34a2ca1749b2b P fe46a8f385dd458f8992090bf3a66bb7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:54:30.588140 13748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:30.574140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46707 (local address 127.0.97.2:38219)
0929 15:54:30.574396 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:30.574406 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:30.581707 (+ 7301us) server_negotiation.cc:407] Connection header received
0929 15:54:30.581861 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:30.581866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:30.581966 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:30.582064 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:54:30.582672 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:30.583710 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:30.585973 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:30.587138 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:30.587306 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:30.587438 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:30.587755 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 15:54:30.587879 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 15:54:30.588387 13771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:30.575227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38219 (local address 127.0.97.1:46707)
0929 15:54:30.581563 (+ 6336us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:30.581583 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:54:30.581723 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:54:30.582101 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:54:30.582106 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:54:30.582132 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:54:30.582473 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:30.582483 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:30.583895 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:30.583900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:30.585806 (+ 1906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:30.585820 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:30.587426 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:30.587433 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:30.587592 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:30.587616 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 15:54:30.587731 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 15:54:30.587864 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6201,"thread_start_us":5985,"threads_started":1}
I20260929 15:54:30.589437 13551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeab19d5e3e345d7a2f34a2ca1749b2b" candidate_uuid: "fe46a8f385dd458f8992090bf3a66bb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5a89efeefbd4058a871316445d49656" is_pre_election: true
I20260929 15:54:30.590037 13551 raft_consensus.cc:2454] T aeab19d5e3e345d7a2f34a2ca1749b2b P f5a89efeefbd4058a871316445d49656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe46a8f385dd458f8992090bf3a66bb7 in term 0.
I20260929 15:54:30.590330 13762 raft_consensus.cc:504] T aeab19d5e3e345d7a2f34a2ca1749b2b P fe46a8f385dd458f8992090bf3a66bb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfd698f9ed44fbe9054024f5efd7ed5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45497 } } peers { permanent_uuid: "f5a89efeefbd4058a871316445d49656" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38219 } } peers { permanent_uuid: "fe46a8f385dd458f8992090bf3a66bb7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38189 } }
I20260929 15:54:30.591017 13762 leader_election.cc:288] T aeab19d5e3e345d7a2f34a2ca1749b2b P fe46a8f385dd458f8992090bf3a66bb7 [CANDIDATE]: Term 1 election: Requested vote from peers 9bfd698f9ed44fbe9054024f5efd7ed5 (127.0.97.3:45497), f5a89efeefbd4058a871316445d49656 (127.0.97.2:38219)
I20260929 15:54:30.591523 13687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeab19d5e3e345d7a2f34a2ca1749b2b" candidate_uuid: "fe46a8f385dd458f8992090bf3a66bb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfd698f9ed44fbe9054024f5efd7ed5"
I20260929 15:54:30.591737 13687 raft_consensus.cc:3037] T aeab19d5e3e345d7a2f34a2ca1749b2b P 9bfd698f9ed44fbe9054024f5efd7ed5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:54:30.594843 13687 raft_consensus.cc:2454] T aeab19d5e3e345d7a2f34a2ca1749b2b P 9bfd698f9ed44fbe9054024f5efd7ed5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe46a8f385dd458f8992090bf3a66bb7 in term 1.
I20260929 15:54:30.595202 13551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeab19d5e3e345d7a2f34a2ca1749b2b" candidate_uuid: "fe46a8f385dd458f8992090bf3a66bb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5a89efeefbd4058a871316445d49656"
I20260929 15:54:30.595242 13341 leader_election.cc:302] T aeab19d5e3e345d7a2f34a2ca1749b2b P fe46a8f385dd458f8992090bf3a66bb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfd698f9ed44fbe9054024f5efd7ed5, fe46a8f385dd458f8992090bf3a66bb7; no voters:
I20260929 15:54:30.595396 13551 raft_consensus.cc:3037] T aeab19d5e3e345d7a2f34a2ca1749b2b P f5a89efeefbd4058a871316445d49656 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:54:30.595546 13762 raft_consensus.cc:2781] T aeab19d5e3e345d7a2f34a2ca1749b2b P fe46a8f385dd458f8992090bf3a66bb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:54:30.596108 13762 raft_consensus.cc:686] T aeab19d5e3e345d7a2f34a2ca1749b2b P fe46a8f385dd458f8992090bf3a66bb7 [term 1 LEADER]: Becoming Leader. State: Replica: fe46a8f385dd458f8992090bf3a66bb7, State: Running, Role: LEADER
I20260929 15:54:30.596662 13762 consensus_queue.cc:237] T aeab19d5e3e345d7a2f34a2ca1749b2b P fe46a8f385dd458f8992090bf3a66bb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfd698f9ed44fbe9054024f5efd7ed5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45497 } } peers { permanent_uuid: "f5a89efeefbd4058a871316445d49656" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38219 } } peers { permanent_uuid: "fe46a8f385dd458f8992090bf3a66bb7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38189 } }
I20260929 15:54:30.598289 13551 raft_consensus.cc:2454] T aeab19d5e3e345d7a2f34a2ca1749b2b P f5a89efeefbd4058a871316445d49656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe46a8f385dd458f8992090bf3a66bb7 in term 1.
I20260929 15:54:30.607758 13255 catalog_manager.cc:5495] T aeab19d5e3e345d7a2f34a2ca1749b2b P fe46a8f385dd458f8992090bf3a66bb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe46a8f385dd458f8992090bf3a66bb7 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "fe46a8f385dd458f8992090bf3a66bb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfd698f9ed44fbe9054024f5efd7ed5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5a89efeefbd4058a871316445d49656" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe46a8f385dd458f8992090bf3a66bb7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38189 } health_report { overall_health: HEALTHY } } }
I20260929 15:54:30.634774 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:30.628043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38614 (local address 127.0.97.1:38189)
0929 15:54:30.628268 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:30.628275 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:30.628631 (+ 356us) server_negotiation.cc:407] Connection header received
0929 15:54:30.628826 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:30.628831 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:30.628944 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:30.629103 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:54:30.629592 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:30.630682 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:30.632277 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:30.632856 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:30.632980 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:30.633645 (+ 665us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:54:30.634169 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:54:30.634346 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:30.634404 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 15:54:30.634501 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 15:54:30.643018 13551 raft_consensus.cc:1261] T aeab19d5e3e345d7a2f34a2ca1749b2b P f5a89efeefbd4058a871316445d49656 [term 1 FOLLOWER]: Refusing update from remote peer fe46a8f385dd458f8992090bf3a66bb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:54:30.643118 13687 raft_consensus.cc:1261] T aeab19d5e3e345d7a2f34a2ca1749b2b P 9bfd698f9ed44fbe9054024f5efd7ed5 [term 1 FOLLOWER]: Refusing update from remote peer fe46a8f385dd458f8992090bf3a66bb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:54:30.644047 13772 consensus_queue.cc:1035] T aeab19d5e3e345d7a2f34a2ca1749b2b P fe46a8f385dd458f8992090bf3a66bb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5a89efeefbd4058a871316445d49656" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:54:30.644443 13762 consensus_queue.cc:1035] T aeab19d5e3e345d7a2f34a2ca1749b2b P fe46a8f385dd458f8992090bf3a66bb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfd698f9ed44fbe9054024f5efd7ed5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:54:30.659525 13776 mvcc.cc:205] Tried to move back new op lower bound from 7334696020544180224 to 7334696020367978496. Current Snapshot: MvccSnapshot[applied={T|T < 7334696020544180224}]
I20260929 15:54:30.674072 13778 mvcc.cc:205] Tried to move back new op lower bound from 7334696020544180224 to 7334696020367978496. Current Snapshot: MvccSnapshot[applied={T|T < 7334696020544180224}]
I20260929 15:54:30.679770 13254 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:54:30.680101 13254 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:54:30.684516 13254 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 576b6935026a416e9d336bd069ec551e: Sending DeleteTablet for 3 replicas of tablet aeab19d5e3e345d7a2f34a2ca1749b2b
I20260929 15:54:30.685743 13667 tablet_service.cc:1514] Processing DeleteTablet for tablet aeab19d5e3e345d7a2f34a2ca1749b2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:54:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49938
I20260929 15:54:30.689291 13531 tablet_service.cc:1514] Processing DeleteTablet for tablet aeab19d5e3e345d7a2f34a2ca1749b2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:54:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50616
I20260929 15:54:30.691701 13530 tablet_service.cc:1514] Processing DeleteTablet for tablet aeab19d5e3e345d7a2f34a2ca1749b2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:54:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50616
I20260929 15:54:30.692739 13388 tablet_service.cc:1514] Processing DeleteTablet for tablet aeab19d5e3e345d7a2f34a2ca1749b2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:54:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38610
I20260929 15:54:30.701143 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 13293
I20260929 15:54:30.701892 13786 tablet_replica.cc:330] T aeab19d5e3e345d7a2f34a2ca1749b2b P f5a89efeefbd4058a871316445d49656: stopping tablet replica
I20260929 15:54:30.702389 13786 raft_consensus.cc:2229] T aeab19d5e3e345d7a2f34a2ca1749b2b P f5a89efeefbd4058a871316445d49656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:54:30.702944 13786 raft_consensus.cc:2258] T aeab19d5e3e345d7a2f34a2ca1749b2b P f5a89efeefbd4058a871316445d49656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:54:30.703964 13787 tablet_replica.cc:330] T aeab19d5e3e345d7a2f34a2ca1749b2b P fe46a8f385dd458f8992090bf3a66bb7: stopping tablet replica
I20260929 15:54:30.704497 13787 raft_consensus.cc:2229] T aeab19d5e3e345d7a2f34a2ca1749b2b P fe46a8f385dd458f8992090bf3a66bb7 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:54:30.707448 13786 ts_tablet_manager.cc:1905] T aeab19d5e3e345d7a2f34a2ca1749b2b P f5a89efeefbd4058a871316445d49656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:54:30.709287 13785 tablet_replica.cc:330] T aeab19d5e3e345d7a2f34a2ca1749b2b P 9bfd698f9ed44fbe9054024f5efd7ed5: stopping tablet replica
I20260929 15:54:30.709839 13785 raft_consensus.cc:2229] T aeab19d5e3e345d7a2f34a2ca1749b2b P 9bfd698f9ed44fbe9054024f5efd7ed5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:54:30.710654 13785 raft_consensus.cc:2258] T aeab19d5e3e345d7a2f34a2ca1749b2b P 9bfd698f9ed44fbe9054024f5efd7ed5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:54:30.712638 13785 ts_tablet_manager.cc:1905] T aeab19d5e3e345d7a2f34a2ca1749b2b P 9bfd698f9ed44fbe9054024f5efd7ed5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:54:30.713932 13787 raft_consensus.cc:2258] T aeab19d5e3e345d7a2f34a2ca1749b2b P fe46a8f385dd458f8992090bf3a66bb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:54:30.718873 13787 ts_tablet_manager.cc:1905] T aeab19d5e3e345d7a2f34a2ca1749b2b P fe46a8f385dd458f8992090bf3a66bb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:54:30.719671 13785 ts_tablet_manager.cc:1918] T aeab19d5e3e345d7a2f34a2ca1749b2b P 9bfd698f9ed44fbe9054024f5efd7ed5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:54:30.719873 13785 log.cc:1196] T aeab19d5e3e345d7a2f34a2ca1749b2b P 9bfd698f9ed44fbe9054024f5efd7ed5: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-2/wal/wals/aeab19d5e3e345d7a2f34a2ca1749b2b
I20260929 15:54:30.720486 13785 ts_tablet_manager.cc:1939] T aeab19d5e3e345d7a2f34a2ca1749b2b P 9bfd698f9ed44fbe9054024f5efd7ed5: Deleting consensus metadata
I20260929 15:54:30.725934 13240 catalog_manager.cc:4841] TS 9bfd698f9ed44fbe9054024f5efd7ed5 (127.0.97.3:45497): tablet aeab19d5e3e345d7a2f34a2ca1749b2b (table test-table [id=51e25a94f97046628ad5ff29d0038a92]) successfully deleted
I20260929 15:54:30.728001 13786 ts_tablet_manager.cc:1918] T aeab19d5e3e345d7a2f34a2ca1749b2b P f5a89efeefbd4058a871316445d49656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:54:30.728134 13786 log.cc:1196] T aeab19d5e3e345d7a2f34a2ca1749b2b P f5a89efeefbd4058a871316445d49656: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-1/wal/wals/aeab19d5e3e345d7a2f34a2ca1749b2b
I20260929 15:54:30.728580 13786 ts_tablet_manager.cc:1939] T aeab19d5e3e345d7a2f34a2ca1749b2b P f5a89efeefbd4058a871316445d49656: Deleting consensus metadata
I20260929 15:54:30.730154 13242 catalog_manager.cc:4841] TS f5a89efeefbd4058a871316445d49656 (127.0.97.2:38219): tablet aeab19d5e3e345d7a2f34a2ca1749b2b (table test-table [id=51e25a94f97046628ad5ff29d0038a92]) successfully deleted
W20260929 15:54:30.730701 13242 catalog_manager.cc:4808] TS f5a89efeefbd4058a871316445d49656 (127.0.97.2:38219): delete failed for tablet aeab19d5e3e345d7a2f34a2ca1749b2b because the tablet was not found. No further retry: Not found: Tablet not found: aeab19d5e3e345d7a2f34a2ca1749b2b
I20260929 15:54:30.737443 13787 ts_tablet_manager.cc:1918] T aeab19d5e3e345d7a2f34a2ca1749b2b P fe46a8f385dd458f8992090bf3a66bb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:54:30.737718 13787 log.cc:1196] T aeab19d5e3e345d7a2f34a2ca1749b2b P fe46a8f385dd458f8992090bf3a66bb7: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697075657932-388-0/minicluster-data/ts-0/wal/wals/aeab19d5e3e345d7a2f34a2ca1749b2b
I20260929 15:54:30.738315 13787 ts_tablet_manager.cc:1939] T aeab19d5e3e345d7a2f34a2ca1749b2b P fe46a8f385dd458f8992090bf3a66bb7: Deleting consensus metadata
I20260929 15:54:30.740161 13242 catalog_manager.cc:4841] TS fe46a8f385dd458f8992090bf3a66bb7 (127.0.97.1:38189): tablet aeab19d5e3e345d7a2f34a2ca1749b2b (table test-table [id=51e25a94f97046628ad5ff29d0038a92]) successfully deleted
I20260929 15:54:30.761118 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:30.704326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38620 (local address 127.0.97.1:38189)
0929 15:54:30.704509 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:30.704517 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:30.726029 (+ 21512us) server_negotiation.cc:407] Connection header received
0929 15:54:30.726102 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:30.726106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:30.726190 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:30.731173 (+ 4983us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:30.731384 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:30.732805 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:30.742522 (+ 9717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:30.745729 (+ 3207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:30.745875 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:30.753187 (+ 7312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:30.753204 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:30.753233 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:30.753287 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:30.757478 (+ 4191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:30.758303 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:30.758312 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:30.758315 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:30.758477 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:30.758918 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:30.758923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:30.758927 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:30.759346 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:30.759641 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:30.759953 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 15:54:30.760720 (+ 767us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 15:54:30.761829 13448 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:54:30.894593 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 13293
I20260929 15:54:30.912519 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 13456
I20260929 15:54:30.925792 13748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:30.913306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50632 (local address 127.0.97.2:38219)
0929 15:54:30.913535 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:30.913539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:30.913558 (+ 19us) server_negotiation.cc:407] Connection header received
0929 15:54:30.913691 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:30.913694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:30.913767 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:30.913851 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:30.915099 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:30.915812 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:30.917057 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:30.917639 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:30.917775 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:30.920344 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:30.920361 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:30.920385 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:30.920419 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:30.923314 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:30.923984 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:30.923991 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:30.923994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:30.924138 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:30.924472 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:30.924475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:30.924482 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:30.924821 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:30.925032 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:30.925293 (+ 261us) server_negotiation.cc:299] Negotiation successful
0929 15:54:30.925541 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260929 15:54:30.926441 13591 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:54:31.014318 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 13456
I20260929 15:54:31.033494 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 13601
I20260929 15:54:31.048089 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:31.034278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49944 (local address 127.0.97.3:45497)
0929 15:54:31.034498 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:31.034502 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:31.034618 (+ 116us) server_negotiation.cc:407] Connection header received
0929 15:54:31.034772 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:31.034776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:31.034842 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:31.034923 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:31.036512 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:31.037199 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:31.038992 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:31.039390 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:31.039503 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:31.042590 (+ 3087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:31.042606 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:31.042626 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:31.042669 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:31.045305 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:31.046111 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:31.046118 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:31.046120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:31.046233 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:31.046576 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:31.046579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:31.046581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:31.046907 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:31.047133 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:31.047494 (+ 361us) server_negotiation.cc:299] Negotiation successful
0929 15:54:31.047715 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 15:54:31.048715 13727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:54:31.137805 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 13601
I20260929 15:54:31.154670 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 13223
I20260929 15:54:31.155845 13286 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:54:31.266219 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 13223
I20260929 15:54:31.289726 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 15:54:31.624830 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 15:54:31.625027 388 mini_ranger.cc:66] Stopping Ranger...
I20260929 15:54:31.970034 388 mini_ranger.cc:68] Stopped Ranger
2026-09-29 15:54:31.970 UTC [12275] LOG: received smart shutdown request
2026-09-29 15:54:31.973 UTC [12275] LOG: background worker "logical replication launcher" (PID 12284) exited with exit code 1
2026-09-29 15:54:31.973 UTC [12279] LOG: shutting down
2026-09-29 15:54:32.142 UTC [12275] LOG: database system is shut down
2026-09-29T15:54:32Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51515 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:54:32 dist-test-slave-r0ck krb5kdc[13801](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:54:32 dist-test-slave-r0ck krb5kdc[13801](info): set up 2 sockets
Sep 29 15:54:32 dist-test-slave-r0ck krb5kdc[13801](info): commencing operation
krb5kdc: starting...
W20260929 15:54:33.357558 388 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.006s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:54:33 dist-test-slave-r0ck krb5kdc[13801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:54:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:54:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:54:33.577142 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33689
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33689,127.0.97.61:44131,127.0.97.60:41517
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:54:33.898037 13814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:54:33.898298 13814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:54:33.898454 13814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:54:33.905546 13814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:54:33.905666 13814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:54:33.905747 13814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:54:33.905846 13814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:54:33.905927 13814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:54:33.915319 13814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33689,127.0.97.61:44131,127.0.97.60:41517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:54:33.916554 13814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:54:33.917791 13814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:54:33.929075 13822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:33.929145 13819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:33.929119 13820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:54:33.929975 13814 server_base.cc:1034] running on GCE node
I20260929 15:54:33.930707 13814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:54:33.931994 13814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:54:33.933231 13814 hybrid_clock.cc:648] HybridClock initialized: now 1790697273933184 us; error 63 us; skew 500 ppm
I20260929 15:54:33.933596 13814 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:54:33 dist-test-slave-r0ck krb5kdc[13801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:54:33.936676 13814 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:54:33.938114 13814 webserver.cc:460] Webserver started at http://127.0.97.62:37487/ using document root <none> and password file <none>
I20260929 15:54:33.938774 13814 fs_manager.cc:359] Metadata directory not provided
I20260929 15:54:33.938918 13814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:54:33.939216 13814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:54:33.942245 13814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "7591019df79842d48ccd6ebb72a3d605"
format_stamp: "Formatted at 2026-09-29 15:54:33 on dist-test-slave-r0ck"
server_key: "5b7722204b92d57a2f9181868e67b9ac"
server_key_iv: "7f6c9fb8ad7ebb1598e5ecc4d9db67be"
server_key_version: "encryptionkey@0"
I20260929 15:54:33.943051 13814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "7591019df79842d48ccd6ebb72a3d605"
format_stamp: "Formatted at 2026-09-29 15:54:33 on dist-test-slave-r0ck"
server_key: "5b7722204b92d57a2f9181868e67b9ac"
server_key_iv: "7f6c9fb8ad7ebb1598e5ecc4d9db67be"
server_key_version: "encryptionkey@0"
I20260929 15:54:33.947892 13814 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 15:54:33.951180 13829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:33.952396 13814 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 15:54:33.952601 13814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "7591019df79842d48ccd6ebb72a3d605"
format_stamp: "Formatted at 2026-09-29 15:54:33 on dist-test-slave-r0ck"
server_key: "5b7722204b92d57a2f9181868e67b9ac"
server_key_iv: "7f6c9fb8ad7ebb1598e5ecc4d9db67be"
server_key_version: "encryptionkey@0"
I20260929 15:54:33.952795 13814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:54:33.979619 13814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:54:33.983542 13814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:54:33.983834 13814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:54:34.001816 13814 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33689
I20260929 15:54:34.001849 13880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33689 every 8 connection(s)
I20260929 15:54:34.003196 13814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:54:34.006800 13881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:54:34.007989 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 13814
I20260929 15:54:34.008219 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:54:34.008872 388 external_mini_cluster.cc:1428] Setting key 715d080a61b8ff5005bbabaca44d9386
I20260929 15:54:34.013602 13881 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33689 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Sep 29 15:54:34 dist-test-slave-r0ck krb5kdc[13801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260929 15:54:34.037571 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:34.016876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33689 (local address 127.0.0.1:55134)
0929 15:54:34.018200 (+ 1324us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:34.018248 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 15:54:34.019952 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:54:34.020846 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:54:34.020850 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:54:34.021463 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:54:34.021783 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:34.021799 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:34.024173 (+ 2374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:34.024177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:34.025334 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:34.025400 (+ 66us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:34.026770 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:34.026775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:34.026887 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:34.027640 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:54:34.027661 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:54:34.029963 (+ 2302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:54:34.032986 (+ 3023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:34.032998 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:34.033009 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:34.033556 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:34.034108 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:34.034111 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:34.034115 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:34.034293 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:34.035381 (+ 1088us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:54:34.035390 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:54:34.035673 (+ 283us) client_negotiation.cc:769] Sending connection context
0929 15:54:34.035928 (+ 255us) client_negotiation.cc:240] Negotiation successful
0929 15:54:34.036430 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1060,"spinlock_wait_cycles":15232,"thread_start_us":833,"threads_started":1}
I20260929 15:54:34.037607 13884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:34.016876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55134 (local address 127.0.97.62:33689)
0929 15:54:34.017807 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:34.017823 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:34.018383 (+ 560us) server_negotiation.cc:407] Connection header received
0929 15:54:34.020311 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:34.020321 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:34.020604 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:34.020816 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:34.021983 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:34.024006 (+ 2023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:34.025527 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:34.026564 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:34.026754 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:34.030188 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:34.030195 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:34.030206 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:34.030232 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:34.032867 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:34.033715 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:34.033721 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:34.033723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:34.033953 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:34.034440 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:34.034448 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:34.034453 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:34.035147 (+ 694us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:34.035336 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:34.036181 (+ 845us) server_negotiation.cc:299] Negotiation successful
0929 15:54:34.036641 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":678,"thread_start_us":444,"threads_started":1}
I20260929 15:54:34.040678 13881 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44131 } has no permanent_uuid. Determining permanent_uuid...
I20260929 15:54:34.041749 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:34.041274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44131 (local address 127.0.0.1:57434)
0929 15:54:34.041477 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:34.041608 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 15:54:34.041988 13830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44131: connect: Connection refused (error 111)
I20260929 15:54:34.043597 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:34.043151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44131 (local address 127.0.0.1:57440)
0929 15:54:34.043319 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:34.043417 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 15:54:34.043800 13881 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44131: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44131: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260929 15:54:34.060348 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:44131
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33689,127.0.97.61:44131,127.0.97.60:41517
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:54:34.093145 13881 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44131 } attempt: 1
I20260929 15:54:34.094358 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:34.093853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44131 (local address 127.0.0.1:57456)
0929 15:54:34.094075 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:34.094183 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 15:54:34.095362 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:34.094976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44131 (local address 127.0.0.1:57464)
0929 15:54:34.095133 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:34.095214 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 15:54:34.095564 13881 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44131: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44131: connect: Connection refused (error 111)
I20260929 15:54:34.163843 13881 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44131 } attempt: 2
I20260929 15:54:34.165189 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:34.164620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44131 (local address 127.0.0.1:57466)
0929 15:54:34.164902 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:34.165020 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260929 15:54:34.166329 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:34.165886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44131 (local address 127.0.0.1:57482)
0929 15:54:34.166066 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:34.166178 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 15:54:34.166517 13881 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44131: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44131: connect: Connection refused (error 111)
I20260929 15:54:34.257824 13881 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44131 } attempt: 3
I20260929 15:54:34.259116 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:34.258610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44131 (local address 127.0.0.1:57486)
0929 15:54:34.258836 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:34.258943 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 15:54:34.260185 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:34.259783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44131 (local address 127.0.0.1:57498)
0929 15:54:34.259944 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:34.260025 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 15:54:34.260388 13881 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44131: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44131: connect: Connection refused (error 111)
W20260929 15:54:34.374258 13888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:54:34.374526 13888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:54:34.374615 13888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:54:34.381723 13888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:54:34.381816 13888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:54:34.381862 13888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:54:34.381919 13888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:54:34.381963 13888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:54:34.391402 13888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33689,127.0.97.61:44131,127.0.97.60:41517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:44131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:54:34.392566 13888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:54:34.393770 13888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 15:54:34.403753 13881 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44131 } attempt: 4
I20260929 15:54:34.405139 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:34.404424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44131 (local address 127.0.0.1:57512)
0929 15:54:34.404786 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:34.404908 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
W20260929 15:54:34.405529 13893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:34.405799 13894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:54:34.406389 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:34.405998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44131 (local address 127.0.0.1:57526)
0929 15:54:34.406131 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:34.406224 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 15:54:34.406380 13896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:54:34.406612 13888 server_base.cc:1034] running on GCE node
W20260929 15:54:34.406613 13881 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44131: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44131: connect: Connection refused (error 111)
I20260929 15:54:34.407526 13888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:54:34.408963 13888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:54:34.410198 13888 hybrid_clock.cc:648] HybridClock initialized: now 1790697274410164 us; error 56 us; skew 500 ppm
I20260929 15:54:34.410528 13888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:54:34 dist-test-slave-r0ck krb5kdc[13801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:54:34.413628 13888 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260929 15:54:34.415016 13888 webserver.cc:460] Webserver started at http://127.0.97.61:43805/ using document root <none> and password file <none>
I20260929 15:54:34.415694 13888 fs_manager.cc:359] Metadata directory not provided
I20260929 15:54:34.415843 13888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:54:34.416146 13888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:54:34.419148 13888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-1/data/instance:
uuid: "45af35641f96452694ee520b440eb6a9"
format_stamp: "Formatted at 2026-09-29 15:54:34 on dist-test-slave-r0ck"
server_key: "c0c4484ccb276b511f3c5729cb02d9ff"
server_key_iv: "dfefa60689927bf8baec75624fd5f6d8"
server_key_version: "encryptionkey@0"
I20260929 15:54:34.419914 13888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-1/wal/instance:
uuid: "45af35641f96452694ee520b440eb6a9"
format_stamp: "Formatted at 2026-09-29 15:54:34 on dist-test-slave-r0ck"
server_key: "c0c4484ccb276b511f3c5729cb02d9ff"
server_key_iv: "dfefa60689927bf8baec75624fd5f6d8"
server_key_version: "encryptionkey@0"
I20260929 15:54:34.424770 13888 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 15:54:34.428107 13903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:34.429406 13888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 15:54:34.429584 13888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-1/wal
uuid: "45af35641f96452694ee520b440eb6a9"
format_stamp: "Formatted at 2026-09-29 15:54:34 on dist-test-slave-r0ck"
server_key: "c0c4484ccb276b511f3c5729cb02d9ff"
server_key_iv: "dfefa60689927bf8baec75624fd5f6d8"
server_key_version: "encryptionkey@0"
I20260929 15:54:34.429786 13888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:54:34.448272 13888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:54:34.452592 13888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:54:34.452883 13888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:54:34.471133 13888 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:44131
I20260929 15:54:34.471170 13954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:44131 every 8 connection(s)
I20260929 15:54:34.472460 13888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-1/data/info.pb
I20260929 15:54:34.475982 13955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:54:34.479492 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 13888
I20260929 15:54:34.479732 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-1/wal/instance
I20260929 15:54:34.480161 388 external_mini_cluster.cc:1428] Setting key eaee6266e10d417b35167d03e128f3d5
I20260929 15:54:34.482388 13955 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33689 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:54:34 dist-test-slave-r0ck krb5kdc[13801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260929 15:54:34.502756 13884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:34.484894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55148 (local address 127.0.97.62:33689)
0929 15:54:34.485141 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:34.485150 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:34.486359 (+ 1209us) server_negotiation.cc:407] Connection header received
0929 15:54:34.488374 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:34.488383 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:34.488491 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:34.488607 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:34.490043 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:34.491227 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:34.492720 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:34.493539 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:34.493672 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:34.497234 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:34.497239 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:34.497243 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:34.497274 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:34.499861 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:34.500634 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:34.500638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:34.500640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:34.500752 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:34.501141 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:34.501149 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:34.501155 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:34.501492 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:34.501609 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:34.502252 (+ 643us) server_negotiation.cc:299] Negotiation successful
0929 15:54:34.502412 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 15:54:34.503681 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:34.485236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33689 (local address 127.0.0.1:55148)
0929 15:54:34.486177 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:34.486225 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 15:54:34.488117 (+ 1892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:54:34.488801 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:54:34.488809 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:54:34.489294 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:54:34.489876 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:34.489892 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:34.491384 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:34.491388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:34.492572 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:34.492586 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:34.493752 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:34.493757 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:34.493936 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:34.494618 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:54:34.494640 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:54:34.497024 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:54:34.500014 (+ 2990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:34.500022 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:34.500040 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:34.500529 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:34.500848 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:34.500851 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:34.500853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:34.501003 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:34.501596 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:54:34.501602 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:54:34.501949 (+ 347us) client_negotiation.cc:769] Sending connection context
0929 15:54:34.502168 (+ 219us) client_negotiation.cc:240] Negotiation successful
0929 15:54:34.502493 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":395,"threads_started":1}
I20260929 15:54:34.505225 13955 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44131 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:54:34 dist-test-slave-r0ck krb5kdc[13801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 15:54:34.521003 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:34.505999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44131 (local address 127.0.0.1:57542)
0929 15:54:34.506151 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:34.506169 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:54:34.506267 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:54:34.507445 (+ 1178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:54:34.507448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:54:34.507781 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:54:34.507903 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:34.507910 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:34.509693 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:34.509700 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:34.511243 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:34.511260 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:34.512073 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:34.512084 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:34.512333 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:34.512908 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:54:34.512936 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:54:34.515438 (+ 2502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:54:34.518634 (+ 3196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:34.518642 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:34.518647 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:34.518987 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:34.519333 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:34.519339 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:34.519343 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:34.519477 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:34.520062 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:54:34.520066 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:54:34.520194 (+ 128us) client_negotiation.cc:769] Sending connection context
0929 15:54:34.520344 (+ 150us) client_negotiation.cc:240] Negotiation successful
0929 15:54:34.520620 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 15:54:34.521108 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:34.506288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57542 (local address 127.0.97.61:44131)
0929 15:54:34.506976 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:34.506985 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:34.507015 (+ 30us) server_negotiation.cc:407] Connection header received
0929 15:54:34.507067 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:34.507076 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:34.507291 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:34.507436 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:34.508050 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:34.509535 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:34.511388 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:34.511905 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:34.512007 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:34.515649 (+ 3642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:34.515659 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:34.515671 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:34.515701 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:34.518517 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:34.519112 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:34.519127 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:34.519129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:34.519231 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:34.519583 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:34.519590 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:34.519593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:34.519900 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:34.520136 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:34.520476 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 15:54:34.520729 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":501,"thread_start_us":357,"threads_started":1}
I20260929 15:54:34.523380 13955 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41517 } has no permanent_uuid. Determining permanent_uuid...
I20260929 15:54:34.524739 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:34.524239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41517 (local address 127.0.0.1:40134)
0929 15:54:34.524434 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:34.524585 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 15:54:34.525022 13907 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41517: connect: Connection refused (error 111)
I20260929 15:54:34.526517 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:34.526209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41517 (local address 127.0.0.1:40138)
0929 15:54:34.526340 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:34.526420 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 15:54:34.526727 13955 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41517: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41517: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260929 15:54:34.530790 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:41517
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33689,127.0.97.61:44131,127.0.97.60:41517
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:54:34.576041 13955 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41517 } attempt: 1
I20260929 15:54:34.577337 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:34.576872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41517 (local address 127.0.0.1:40152)
0929 15:54:34.577083 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:34.577170 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260929 15:54:34.578452 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:34.578093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41517 (local address 127.0.0.1:40158)
0929 15:54:34.578254 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:34.578338 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 15:54:34.578694 13955 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41517: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41517: connect: Connection refused (error 111)
I20260929 15:54:34.646886 13955 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41517 } attempt: 2
I20260929 15:54:34.648011 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:34.647488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41517 (local address 127.0.0.1:40172)
0929 15:54:34.647712 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:34.647823 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 15:54:34.649046 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:34.648664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41517 (local address 127.0.0.1:40174)
0929 15:54:34.648825 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:34.648909 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 15:54:34.649305 13955 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41517: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41517: connect: Connection refused (error 111)
I20260929 15:54:34.705865 13881 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44131 } attempt: 5
Sep 29 15:54:34 dist-test-slave-r0ck krb5kdc[13801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 15:54:34.720683 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:34.706617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44131 (local address 127.0.0.1:57558)
0929 15:54:34.706839 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:34.706857 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:54:34.706962 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:54:34.707342 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:54:34.707345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:54:34.707705 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:54:34.707877 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:34.707885 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:34.709458 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:34.709462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:34.710356 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:34.710367 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:34.711544 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:34.711549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:34.711702 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:34.712384 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:54:34.712401 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:54:34.714411 (+ 2010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:54:34.718423 (+ 4012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:34.718427 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:34.718429 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:34.718780 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:34.719181 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:34.719185 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:34.719187 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:34.719306 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:34.719898 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:54:34.719902 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:54:34.720038 (+ 136us) client_negotiation.cc:769] Sending connection context
0929 15:54:34.720164 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 15:54:34.720391 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 15:54:34.720813 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:34.706719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57558 (local address 127.0.97.61:44131)
0929 15:54:34.706923 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:34.706929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:34.706952 (+ 23us) server_negotiation.cc:407] Connection header received
0929 15:54:34.707113 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:34.707118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:34.707198 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:34.707308 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:34.708042 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:34.709302 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:34.710516 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:34.711376 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:34.711532 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:34.714636 (+ 3104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:34.714644 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:34.714651 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:34.714698 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:34.718281 (+ 3583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:34.718905 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:34.718911 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:34.718915 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:34.719058 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:34.719405 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:34.719409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:34.719412 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:34.719757 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:34.719905 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:34.720295 (+ 390us) server_negotiation.cc:299] Negotiation successful
0929 15:54:34.720512 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 15:54:34.721916 13881 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41517 } has no permanent_uuid. Determining permanent_uuid...
I20260929 15:54:34.722922 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:34.722538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41517 (local address 127.0.0.1:40190)
0929 15:54:34.722705 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:34.722788 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 15:54:34.723848 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:34.723551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41517 (local address 127.0.0.1:40200)
0929 15:54:34.723682 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:34.723759 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 15:54:34.724072 13881 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41517: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41517: connect: Connection refused (error 111)
I20260929 15:54:34.740578 13955 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41517 } attempt: 3
I20260929 15:54:34.741811 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:34.741209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41517 (local address 127.0.0.1:40208)
0929 15:54:34.741476 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:34.741596 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 15:54:34.742914 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:34.742447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41517 (local address 127.0.0.1:40210)
0929 15:54:34.742685 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:34.742773 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260929 15:54:34.743141 13955 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41517: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41517: connect: Connection refused (error 111)
I20260929 15:54:34.775405 13881 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41517 } attempt: 1
I20260929 15:54:34.776508 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:34.776042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41517 (local address 127.0.0.1:40216)
0929 15:54:34.776235 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:34.776331 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 15:54:34.777498 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:34.777100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41517 (local address 127.0.0.1:40232)
0929 15:54:34.777270 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:34.777335 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260929 15:54:34.777709 13881 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41517: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41517: connect: Connection refused (error 111)
I20260929 15:54:34.845938 13881 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41517 } attempt: 2
I20260929 15:54:34.847103 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:34.846544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41517 (local address 127.0.0.1:40242)
0929 15:54:34.846775 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:34.846901 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 15:54:34.848062 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:34.847735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41517 (local address 127.0.0.1:40256)
0929 15:54:34.847869 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:34.847953 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 15:54:34.848278 13881 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41517: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41517: connect: Connection refused (error 111)
W20260929 15:54:34.848479 13962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:54:34.848780 13962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:54:34.848938 13962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:54:34.856124 13962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:54:34.856227 13962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:54:34.856344 13962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:54:34.856432 13962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:54:34.856508 13962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:54:34.866261 13962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33689,127.0.97.61:44131,127.0.97.60:41517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:41517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:54:34.867496 13962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:54:34.868839 13962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:54:34.880254 13967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:34.880321 13970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:34.880286 13968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:54:34.881188 13962 server_base.cc:1034] running on GCE node
I20260929 15:54:34.881918 13962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:54:34.883209 13962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:54:34.884431 13962 hybrid_clock.cc:648] HybridClock initialized: now 1790697274884398 us; error 56 us; skew 500 ppm
I20260929 15:54:34.884760 13962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:54:34.886361 13955 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41517 } attempt: 4
Sep 29 15:54:34 dist-test-slave-r0ck krb5kdc[13801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:54:34.887392 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:34.886912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41517 (local address 127.0.0.1:40268)
0929 15:54:34.887144 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:34.887221 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 15:54:34.887825 13962 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260929 15:54:34.888674 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:34.888139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41517 (local address 127.0.0.1:40284)
0929 15:54:34.888364 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:34.888486 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260929 15:54:34.888995 13955 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41517: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41517: connect: Connection refused (error 111)
I20260929 15:54:34.889894 13962 webserver.cc:460] Webserver started at http://127.0.97.60:40767/ using document root <none> and password file <none>
I20260929 15:54:34.890475 13962 fs_manager.cc:359] Metadata directory not provided
I20260929 15:54:34.890568 13962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:54:34.890785 13962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:54:34.893548 13962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-2/data/instance:
uuid: "dd0745debd414fe386785de6c3c00e04"
format_stamp: "Formatted at 2026-09-29 15:54:34 on dist-test-slave-r0ck"
server_key: "e0d22a87f2607ce70a6fa863f7e57afd"
server_key_iv: "55dcadcaf9577e46219d9c2a28292e6b"
server_key_version: "encryptionkey@0"
I20260929 15:54:34.894273 13962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-2/wal/instance:
uuid: "dd0745debd414fe386785de6c3c00e04"
format_stamp: "Formatted at 2026-09-29 15:54:34 on dist-test-slave-r0ck"
server_key: "e0d22a87f2607ce70a6fa863f7e57afd"
server_key_iv: "55dcadcaf9577e46219d9c2a28292e6b"
server_key_version: "encryptionkey@0"
I20260929 15:54:34.899080 13962 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 15:54:34.902151 13977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:34.903378 13962 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260929 15:54:34.903534 13962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-2/wal
uuid: "dd0745debd414fe386785de6c3c00e04"
format_stamp: "Formatted at 2026-09-29 15:54:34 on dist-test-slave-r0ck"
server_key: "e0d22a87f2607ce70a6fa863f7e57afd"
server_key_iv: "55dcadcaf9577e46219d9c2a28292e6b"
server_key_version: "encryptionkey@0"
I20260929 15:54:34.903730 13962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:54:34.927783 13962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:54:34.931744 13962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:54:34.932019 13962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:54:34.949901 13962 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:41517
I20260929 15:54:34.949921 14028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:41517 every 8 connection(s)
I20260929 15:54:34.951148 13962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-2/data/info.pb
I20260929 15:54:34.954593 14029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:54:34.954506 13881 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41517 } attempt: 3
I20260929 15:54:34.960239 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 13962
I20260929 15:54:34.960445 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697075657932-388-0/minicluster-data/master-2/wal/instance
I20260929 15:54:34.961022 388 external_mini_cluster.cc:1428] Setting key caf800add84a56cd20458249ddcf50d7
I20260929 15:54:34.965570 14029 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33689 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:54:34 dist-test-slave-r0ck krb5kdc[13801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 29 15:54:34 dist-test-slave-r0ck krb5kdc[13801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:54:34.979454 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:34.955169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41517 (local address 127.0.0.1:40288)
0929 15:54:34.955406 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:34.955429 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 15:54:34.955556 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:54:34.959069 (+ 3513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:54:34.959074 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:54:34.959562 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:54:34.959771 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:34.959784 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:34.962262 (+ 2478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:34.962272 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:34.964237 (+ 1965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:34.964256 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:34.966064 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:34.966073 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:34.966352 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:34.967173 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:54:34.967205 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:54:34.970742 (+ 3537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:54:34.976412 (+ 5670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:34.976418 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:34.976423 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:34.976947 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:34.977513 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:34.977518 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:34.977521 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:34.977659 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:34.978423 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:54:34.978427 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:54:34.978608 (+ 181us) client_negotiation.cc:769] Sending connection context
0929 15:54:34.978755 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 15:54:34.979046 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
Sep 29 15:54:34 dist-test-slave-r0ck krb5kdc[13801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 15:54:34.980783 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:34.955944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40288 (local address 127.0.97.60:41517)
0929 15:54:34.956627 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:34.956638 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:34.956676 (+ 38us) server_negotiation.cc:407] Connection header received
0929 15:54:34.958377 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:34.958396 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:34.958737 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:34.959044 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:34.959957 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:34.962028 (+ 2071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:34.964479 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:34.965842 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:34.966801 (+ 959us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:34.971047 (+ 4246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:34.971060 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:34.971079 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:34.971137 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:34.976270 (+ 5133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:34.977084 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:34.977102 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:34.977112 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:34.977337 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:34.977774 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:34.977779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:34.977782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:34.978285 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:34.978480 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:34.979006 (+ 526us) server_negotiation.cc:299] Negotiation successful
0929 15:54:34.979428 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":284,"threads_started":1}
I20260929 15:54:34.987274 13884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:34.970439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55154 (local address 127.0.97.62:33689)
0929 15:54:34.970599 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:34.970606 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:34.971154 (+ 548us) server_negotiation.cc:407] Connection header received
0929 15:54:34.971415 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:34.971419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:34.971491 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:34.971597 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:34.973213 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:34.974173 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:34.975530 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:34.975969 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:34.976101 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:34.980223 (+ 4122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:34.980229 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:34.980235 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:34.980272 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:34.983900 (+ 3628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:34.984900 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:34.984908 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:34.984914 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:34.985098 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:34.985543 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:34.985548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:34.985551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:34.985967 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:34.986133 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:34.986694 (+ 561us) server_negotiation.cc:299] Negotiation successful
0929 15:54:34.986891 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 15:54:34.987314 14034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:34.970367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33689 (local address 127.0.0.1:55154)
0929 15:54:34.971021 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:34.971065 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 15:54:34.971262 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:54:34.971643 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:54:34.971649 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:54:34.972648 (+ 999us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:54:34.973056 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:34.973076 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:34.974329 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:34.974333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:34.975391 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:34.975403 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:34.976180 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:34.976188 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:34.976553 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:34.977258 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:54:34.977285 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:54:34.980031 (+ 2746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:54:34.984041 (+ 4010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:34.984054 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:34.984062 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:34.984766 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:34.985216 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:34.985221 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:34.985224 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:34.985420 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:34.986103 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:54:34.986112 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:54:34.986350 (+ 238us) client_negotiation.cc:769] Sending connection context
0929 15:54:34.986566 (+ 216us) client_negotiation.cc:240] Negotiation successful
0929 15:54:34.986881 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":134,"spinlock_wait_cycles":7552,"thread_start_us":311,"threads_started":1}
I20260929 15:54:34.988428 13881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7591019df79842d48ccd6ebb72a3d605: Bootstrap starting.
I20260929 15:54:34.988569 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:34.962000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40302 (local address 127.0.97.60:41517)
0929 15:54:34.969998 (+ 7998us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:34.970003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:34.970024 (+ 21us) server_negotiation.cc:407] Connection header received
0929 15:54:34.970069 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:34.970073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:34.970138 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:34.970244 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:34.972292 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:34.973577 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:34.975587 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:34.976372 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:34.976492 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:34.981621 (+ 5129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:34.981628 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:34.981633 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:34.981665 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:34.984862 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:34.985639 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:34.985643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:34.985646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:34.985775 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:34.986213 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:34.986218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:34.986222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:34.986602 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:34.986871 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:34.987943 (+ 1072us) server_negotiation.cc:299] Negotiation successful
0929 15:54:34.988164 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":7857,"spinlock_wait_cycles":52096,"thread_start_us":7731,"threads_started":1}
I20260929 15:54:34.989260 14029 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44131 } has no permanent_uuid. Determining permanent_uuid...
I20260929 15:54:34.992209 13881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7591019df79842d48ccd6ebb72a3d605: Neither blocks nor log segments found. Creating new log.
I20260929 15:54:34.993525 13881 log.cc:824] T 00000000000000000000000000000000 P 7591019df79842d48ccd6ebb72a3d605: Log is configured to *not* fsync() on all Append() calls
I20260929 15:54:34.996348 13881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7591019df79842d48ccd6ebb72a3d605: No bootstrap required, opened a new log
Sep 29 15:54:34 dist-test-slave-r0ck krb5kdc[13801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 29 15:54:34 dist-test-slave-r0ck krb5kdc[13801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:54:35.005268 14034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:34.989968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44131 (local address 127.0.0.1:57566)
0929 15:54:34.990226 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:34.990243 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:54:34.990349 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:54:34.990803 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:54:34.990808 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:54:34.991220 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:54:34.991385 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:34.991396 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:34.992790 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:34.992796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:34.993807 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:34.993820 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:34.994801 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:34.994807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:34.995035 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:34.995680 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:54:34.995700 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:54:34.998234 (+ 2534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:54:35.001843 (+ 3609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:35.001850 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:35.001855 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:35.002602 (+ 747us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:35.003075 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:35.003080 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:35.003084 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:35.003583 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:35.004283 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:54:35.004290 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:54:35.004456 (+ 166us) client_negotiation.cc:769] Sending connection context
0929 15:54:35.004582 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 15:54:35.004875 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260929 15:54:35.006613 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:34.990154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57566 (local address 127.0.97.61:44131)
0929 15:54:34.990358 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:34.990364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:34.990385 (+ 21us) server_negotiation.cc:407] Connection header received
0929 15:54:34.990447 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:34.990452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:34.990532 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:34.990636 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:34.991561 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:34.992655 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:34.993939 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:34.994638 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:34.994774 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:34.998439 (+ 3665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:34.998446 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:34.998452 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:34.998497 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:35.001687 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:35.002755 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:35.002761 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:35.002765 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:35.002930 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:35.003751 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:35.003758 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:35.003762 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:35.004118 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:35.005828 (+ 1710us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:35.005975 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 15:54:35.006226 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 15:54:35.008015 14029 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41517 } has no permanent_uuid. Determining permanent_uuid...
I20260929 15:54:35.009114 13884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:34.990872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55170 (local address 127.0.97.62:33689)
0929 15:54:34.991025 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:34.991029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:34.991041 (+ 12us) server_negotiation.cc:407] Connection header received
0929 15:54:34.991077 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:34.991080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:34.991126 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:34.991232 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:34.992678 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:34.993838 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:34.995527 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:34.996243 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:34.996366 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:35.001369 (+ 5003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:35.001378 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:35.001382 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:35.001424 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:35.005249 (+ 3825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:35.006193 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:35.006198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:35.006201 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:35.006337 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:35.006813 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:35.006818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:35.006821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:35.007210 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:35.007492 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:35.008605 (+ 1113us) server_negotiation.cc:299] Negotiation successful
0929 15:54:35.008797 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 15:54:35.011642 13881 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7591019df79842d48ccd6ebb72a3d605 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7591019df79842d48ccd6ebb72a3d605" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33689 } } peers { permanent_uuid: "45af35641f96452694ee520b440eb6a9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44131 } } peers { permanent_uuid: "dd0745debd414fe386785de6c3c00e04" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41517 } }
I20260929 15:54:35.012018 13881 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7591019df79842d48ccd6ebb72a3d605 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:54:35.012146 13881 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7591019df79842d48ccd6ebb72a3d605 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7591019df79842d48ccd6ebb72a3d605, State: Initialized, Role: FOLLOWER
I20260929 15:54:35.012842 13881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7591019df79842d48ccd6ebb72a3d605 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7591019df79842d48ccd6ebb72a3d605" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33689 } } peers { permanent_uuid: "45af35641f96452694ee520b440eb6a9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44131 } } peers { permanent_uuid: "dd0745debd414fe386785de6c3c00e04" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41517 } }
Sep 29 15:54:35 dist-test-slave-r0ck krb5kdc[13801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 29 15:54:35 dist-test-slave-r0ck krb5kdc[13801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 15:54:35.022590 14036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7591019df79842d48ccd6ebb72a3d605 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7591019df79842d48ccd6ebb72a3d605" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33689 } } peers { permanent_uuid: "45af35641f96452694ee520b440eb6a9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44131 } } peers { permanent_uuid: "dd0745debd414fe386785de6c3c00e04" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41517 } } }
I20260929 15:54:35.022994 14036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7591019df79842d48ccd6ebb72a3d605 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:54:35.024102 13881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7591019df79842d48ccd6ebb72a3d605 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:54:35.026327 14034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:35.009034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41517 (local address 127.0.0.1:40314)
0929 15:54:35.009280 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:35.009297 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:54:35.009418 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:54:35.010096 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:54:35.010101 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:54:35.010624 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:54:35.010778 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:35.010787 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:35.012478 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:35.012483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:35.014033 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:35.014050 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:35.014995 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:35.015001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:35.015169 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:35.015868 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:54:35.015896 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:54:35.019095 (+ 3199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:54:35.023374 (+ 4279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:35.023381 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:35.023384 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:35.023878 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:35.024315 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:35.024322 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:35.024325 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:35.024474 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:35.025225 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:54:35.025229 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:54:35.025420 (+ 191us) client_negotiation.cc:769] Sending connection context
0929 15:54:35.025589 (+ 169us) client_negotiation.cc:240] Negotiation successful
0929 15:54:35.025922 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150}
I20260929 15:54:35.027102 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:35.008906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40314 (local address 127.0.97.60:41517)
0929 15:54:35.009114 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:35.009120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:35.009689 (+ 569us) server_negotiation.cc:407] Connection header received
0929 15:54:35.009813 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:35.009817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:35.009922 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:35.010936 (+ 1014us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:35.011071 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:35.012337 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:35.014198 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:35.014834 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:35.017321 (+ 2487us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:35.019312 (+ 1991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:35.019321 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:35.019328 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:35.019370 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:35.023243 (+ 3873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:35.024033 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:35.024039 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:35.024044 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:35.024214 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:35.024588 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:35.024593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:35.024596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:35.024882 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:35.025068 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:35.026752 (+ 1684us) server_negotiation.cc:299] Negotiation successful
0929 15:54:35.026898 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 15:54:35.031412 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:35.011470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57574 (local address 127.0.97.61:44131)
0929 15:54:35.011633 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:35.011638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:35.012007 (+ 369us) server_negotiation.cc:407] Connection header received
0929 15:54:35.012066 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:35.012071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:35.012145 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:35.012228 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:35.014182 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:35.015265 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:35.017204 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:35.017733 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:35.017807 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:35.023316 (+ 5509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:35.023323 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:35.023327 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:35.023376 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:35.027201 (+ 3825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:35.028319 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:35.028327 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:35.028334 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:35.028535 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:35.029026 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:35.029031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:35.029035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:35.029416 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:35.029695 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:35.030837 (+ 1142us) server_negotiation.cc:299] Negotiation successful
0929 15:54:35.031075 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 15:54:35.032132 14047 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 15:54:35.032871 14029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd0745debd414fe386785de6c3c00e04: Bootstrap starting.
I20260929 15:54:35.036214 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:54:35.037031 14029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd0745debd414fe386785de6c3c00e04: Neither blocks nor log segments found. Creating new log.
I20260929 15:54:35.038321 14029 log.cc:824] T 00000000000000000000000000000000 P dd0745debd414fe386785de6c3c00e04: Log is configured to *not* fsync() on all Append() calls
I20260929 15:54:35.047695 14029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd0745debd414fe386785de6c3c00e04: No bootstrap required, opened a new log
I20260929 15:54:35.037360 13814 master_runner.cc:186] Error getting master registration for 127.0.97.61:44131: OK, instance_id { permanent_uuid: "45af35641f96452694ee520b440eb6a9" instance_seqno: 1790697274466048 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 15:54:35.049239 13814 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:54:35.057824 14029 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd0745debd414fe386785de6c3c00e04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7591019df79842d48ccd6ebb72a3d605" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33689 } } peers { permanent_uuid: "45af35641f96452694ee520b440eb6a9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44131 } } peers { permanent_uuid: "dd0745debd414fe386785de6c3c00e04" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41517 } }
I20260929 15:54:35.058388 14029 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd0745debd414fe386785de6c3c00e04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:54:35.058525 14029 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd0745debd414fe386785de6c3c00e04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd0745debd414fe386785de6c3c00e04, State: Initialized, Role: FOLLOWER
I20260929 15:54:35.059441 14029 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd0745debd414fe386785de6c3c00e04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7591019df79842d48ccd6ebb72a3d605" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33689 } } peers { permanent_uuid: "45af35641f96452694ee520b440eb6a9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44131 } } peers { permanent_uuid: "dd0745debd414fe386785de6c3c00e04" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41517 } }
I20260929 15:54:35.060796 14056 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd0745debd414fe386785de6c3c00e04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7591019df79842d48ccd6ebb72a3d605" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33689 } } peers { permanent_uuid: "45af35641f96452694ee520b440eb6a9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44131 } } peers { permanent_uuid: "dd0745debd414fe386785de6c3c00e04" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41517 } } }
I20260929 15:54:35.061195 14056 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd0745debd414fe386785de6c3c00e04 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:54:35.062296 14029 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd0745debd414fe386785de6c3c00e04 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:54:35.065950 13884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:35.048206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55184 (local address 127.0.97.62:33689)
0929 15:54:35.048420 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:35.048425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:35.048449 (+ 24us) server_negotiation.cc:407] Connection header received
0929 15:54:35.048511 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:35.048514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:35.048574 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:35.048695 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:35.051215 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:35.052307 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:35.054877 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:35.055297 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:35.055438 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:35.058976 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:35.058982 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:35.058987 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:35.059038 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:35.062738 (+ 3700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:35.063666 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:35.063672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:35.063676 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:35.063831 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:35.064272 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:35.064278 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:35.064281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:35.064668 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:35.064908 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:35.065316 (+ 408us) server_negotiation.cc:299] Negotiation successful
0929 15:54:35.065564 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 15:54:35.074213 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:35.048176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57590 (local address 127.0.97.61:44131)
0929 15:54:35.048347 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:35.048353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:35.054763 (+ 6410us) server_negotiation.cc:407] Connection header received
0929 15:54:35.054944 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:35.054961 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:35.055045 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:35.055118 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:35.058275 (+ 3157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:35.059341 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:35.062276 (+ 2935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:35.062976 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:35.063099 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:35.066621 (+ 3522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:35.066628 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:35.066634 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:35.066675 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:35.071353 (+ 4678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:35.072128 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:35.072134 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:35.072138 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:35.072273 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:35.072675 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:35.072680 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:35.072683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:35.072963 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:35.073231 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:35.073585 (+ 354us) server_negotiation.cc:299] Negotiation successful
0929 15:54:35.073818 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 15:54:35.077329 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:35.053749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40320 (local address 127.0.97.60:41517)
0929 15:54:35.053902 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:35.053908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:35.059765 (+ 5857us) server_negotiation.cc:407] Connection header received
0929 15:54:35.059932 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:35.059938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:35.060015 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:35.060109 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:35.063036 (+ 2927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:35.063854 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:35.066592 (+ 2738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:35.067282 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:35.067404 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:35.069967 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:35.069972 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:35.069975 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:35.070004 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:35.073587 (+ 3583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:35.074629 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:35.074638 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:35.074644 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:35.074827 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:35.075999 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:35.076002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:35.076004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:35.076292 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:35.076505 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:35.076913 (+ 408us) server_negotiation.cc:299] Negotiation successful
0929 15:54:35.077086 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":36}
W20260929 15:54:35.092502 14067 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dd0745debd414fe386785de6c3c00e04: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 15:54:35.092684 14067 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 15:54:35.087666 13962 master_runner.cc:186] Error getting master registration for 127.0.97.61:44131: OK, instance_id { permanent_uuid: "45af35641f96452694ee520b440eb6a9" instance_seqno: 1790697274466048 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 15:54:35.098330 13962 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:54:35.179188 14036 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7591019df79842d48ccd6ebb72a3d605 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:54:35.179545 14036 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7591019df79842d48ccd6ebb72a3d605 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7591019df79842d48ccd6ebb72a3d605" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33689 } } peers { permanent_uuid: "45af35641f96452694ee520b440eb6a9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44131 } } peers { permanent_uuid: "dd0745debd414fe386785de6c3c00e04" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41517 } }
I20260929 15:54:35.180896 14036 leader_election.cc:288] T 00000000000000000000000000000000 P 7591019df79842d48ccd6ebb72a3d605 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45af35641f96452694ee520b440eb6a9 (127.0.97.61:44131), dd0745debd414fe386785de6c3c00e04 (127.0.97.60:41517)
I20260929 15:54:35.181809 14004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7591019df79842d48ccd6ebb72a3d605" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd0745debd414fe386785de6c3c00e04" is_pre_election: true
I20260929 15:54:35.182422 14004 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dd0745debd414fe386785de6c3c00e04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7591019df79842d48ccd6ebb72a3d605 in term 0.
I20260929 15:54:35.183069 13833 leader_election.cc:302] T 00000000000000000000000000000000 P 7591019df79842d48ccd6ebb72a3d605 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7591019df79842d48ccd6ebb72a3d605, dd0745debd414fe386785de6c3c00e04; no voters:
I20260929 15:54:35.183640 14036 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7591019df79842d48ccd6ebb72a3d605 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:54:35.183781 14036 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7591019df79842d48ccd6ebb72a3d605 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:54:35.183904 14036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7591019df79842d48ccd6ebb72a3d605 [term 0 FOLLOWER]: Advancing to term 1
W20260929 15:54:35.184325 13983 tablet.cc:2369] T 00000000000000000000000000000000 P dd0745debd414fe386785de6c3c00e04: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:54:35.187100 14036 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7591019df79842d48ccd6ebb72a3d605 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7591019df79842d48ccd6ebb72a3d605" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33689 } } peers { permanent_uuid: "45af35641f96452694ee520b440eb6a9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44131 } } peers { permanent_uuid: "dd0745debd414fe386785de6c3c00e04" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41517 } }
I20260929 15:54:35.187742 14036 leader_election.cc:288] T 00000000000000000000000000000000 P 7591019df79842d48ccd6ebb72a3d605 [CANDIDATE]: Term 1 election: Requested vote from peers 45af35641f96452694ee520b440eb6a9 (127.0.97.61:44131), dd0745debd414fe386785de6c3c00e04 (127.0.97.60:41517)
I20260929 15:54:35.188189 14004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7591019df79842d48ccd6ebb72a3d605" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd0745debd414fe386785de6c3c00e04"
I20260929 15:54:35.188232 13955 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41517 } attempt: 5
I20260929 15:54:35.188397 14004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd0745debd414fe386785de6c3c00e04 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:54:35.191771 14004 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dd0745debd414fe386785de6c3c00e04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7591019df79842d48ccd6ebb72a3d605 in term 1.
I20260929 15:54:35.192165 13833 leader_election.cc:302] T 00000000000000000000000000000000 P 7591019df79842d48ccd6ebb72a3d605 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7591019df79842d48ccd6ebb72a3d605, dd0745debd414fe386785de6c3c00e04; no voters:
I20260929 15:54:35.192565 14036 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7591019df79842d48ccd6ebb72a3d605 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:54:35.193192 14036 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7591019df79842d48ccd6ebb72a3d605 [term 1 LEADER]: Becoming Leader. State: Replica: 7591019df79842d48ccd6ebb72a3d605, State: Running, Role: LEADER
I20260929 15:54:35.193830 14036 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7591019df79842d48ccd6ebb72a3d605 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7591019df79842d48ccd6ebb72a3d605" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33689 } } peers { permanent_uuid: "45af35641f96452694ee520b440eb6a9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44131 } } peers { permanent_uuid: "dd0745debd414fe386785de6c3c00e04" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41517 } }
Sep 29 15:54:35 dist-test-slave-r0ck krb5kdc[13801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 15:54:35.198748 14069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7591019df79842d48ccd6ebb72a3d605 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7591019df79842d48ccd6ebb72a3d605. Latest consensus state: current_term: 1 leader_uuid: "7591019df79842d48ccd6ebb72a3d605" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7591019df79842d48ccd6ebb72a3d605" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33689 } } peers { permanent_uuid: "45af35641f96452694ee520b440eb6a9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44131 } } peers { permanent_uuid: "dd0745debd414fe386785de6c3c00e04" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41517 } } }
I20260929 15:54:35.199162 14069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7591019df79842d48ccd6ebb72a3d605 [sys.catalog]: This master's current role is: LEADER
I20260929 15:54:35.203454 14071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:54:35.205747 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:35.188744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41517 (local address 127.0.0.1:40322)
0929 15:54:35.188935 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:35.188961 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 15:54:35.189070 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:54:35.190050 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:54:35.190056 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:54:35.190583 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:54:35.190783 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:35.190794 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:35.192195 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:35.192202 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:35.193653 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:35.193665 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:35.194697 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:35.194705 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:35.194908 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:35.195632 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:54:35.195654 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:54:35.198501 (+ 2847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:54:35.202710 (+ 4209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:35.202715 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:35.202719 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:35.203225 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:35.203666 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:35.203671 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:35.203675 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:35.203837 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:35.204642 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:54:35.204647 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:54:35.204801 (+ 154us) client_negotiation.cc:769] Sending connection context
0929 15:54:35.204955 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 15:54:35.205278 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260929 15:54:35.207633 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:35.189509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40322 (local address 127.0.97.60:41517)
0929 15:54:35.189714 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:35.189721 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:35.189743 (+ 22us) server_negotiation.cc:407] Connection header received
0929 15:54:35.189792 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:35.189797 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:35.189886 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:35.189991 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:35.190950 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:35.192086 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:35.193813 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:35.194533 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:35.194648 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:35.198727 (+ 4079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:35.198734 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:35.198738 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:35.198777 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:35.202536 (+ 3759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:35.203341 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:35.203346 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:35.203350 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:35.203489 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:35.203981 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:35.204009 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:35.204013 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:35.204400 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:35.204543 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:35.206287 (+ 1744us) server_negotiation.cc:299] Negotiation successful
0929 15:54:35.206753 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 15:54:35.208863 14071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:54:35.213107 13955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45af35641f96452694ee520b440eb6a9: Bootstrap starting.
I20260929 15:54:35.222715 13955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45af35641f96452694ee520b440eb6a9: Neither blocks nor log segments found. Creating new log.
I20260929 15:54:35.222925 14004 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dd0745debd414fe386785de6c3c00e04 [term 1 FOLLOWER]: Refusing update from remote peer 7591019df79842d48ccd6ebb72a3d605: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:54:35.224042 13955 log.cc:824] T 00000000000000000000000000000000 P 45af35641f96452694ee520b440eb6a9: Log is configured to *not* fsync() on all Append() calls
I20260929 15:54:35.223953 14036 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7591019df79842d48ccd6ebb72a3d605 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd0745debd414fe386785de6c3c00e04" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:54:35.233570 13955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45af35641f96452694ee520b440eb6a9: No bootstrap required, opened a new log
W20260929 15:54:35.236625 13835 tablet.cc:2369] T 00000000000000000000000000000000 P 7591019df79842d48ccd6ebb72a3d605: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:54:35.244820 13955 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45af35641f96452694ee520b440eb6a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7591019df79842d48ccd6ebb72a3d605" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33689 } } peers { permanent_uuid: "45af35641f96452694ee520b440eb6a9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44131 } } peers { permanent_uuid: "dd0745debd414fe386785de6c3c00e04" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41517 } }
I20260929 15:54:35.245224 13955 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45af35641f96452694ee520b440eb6a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:54:35.245354 13955 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45af35641f96452694ee520b440eb6a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45af35641f96452694ee520b440eb6a9, State: Initialized, Role: FOLLOWER
I20260929 15:54:35.246004 13955 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45af35641f96452694ee520b440eb6a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7591019df79842d48ccd6ebb72a3d605" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33689 } } peers { permanent_uuid: "45af35641f96452694ee520b440eb6a9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44131 } } peers { permanent_uuid: "dd0745debd414fe386785de6c3c00e04" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41517 } }
I20260929 15:54:35.247563 14076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45af35641f96452694ee520b440eb6a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7591019df79842d48ccd6ebb72a3d605" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33689 } } peers { permanent_uuid: "45af35641f96452694ee520b440eb6a9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44131 } } peers { permanent_uuid: "dd0745debd414fe386785de6c3c00e04" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41517 } } }
I20260929 15:54:35.248559 13955 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45af35641f96452694ee520b440eb6a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:54:35.248586 13930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7591019df79842d48ccd6ebb72a3d605" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45af35641f96452694ee520b440eb6a9" is_pre_election: true
I20260929 15:54:35.249166 13930 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 45af35641f96452694ee520b440eb6a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7591019df79842d48ccd6ebb72a3d605 in term 0.
I20260929 15:54:35.249555 13929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7591019df79842d48ccd6ebb72a3d605" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45af35641f96452694ee520b440eb6a9"
I20260929 15:54:35.249786 13929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45af35641f96452694ee520b440eb6a9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:54:35.254216 13929 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 45af35641f96452694ee520b440eb6a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7591019df79842d48ccd6ebb72a3d605 in term 1.
I20260929 15:54:35.255162 13928 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 45af35641f96452694ee520b440eb6a9 [term 1 FOLLOWER]: Refusing update from remote peer 7591019df79842d48ccd6ebb72a3d605: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:54:35.257560 14036 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7591019df79842d48ccd6ebb72a3d605 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45af35641f96452694ee520b440eb6a9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:54:35.260427 14073 mvcc.cc:205] Tried to move back new op lower bound from 7334696039304081408 to 7334696039198863360. Current Snapshot: MvccSnapshot[applied={T|T < 7334696039304081408}]
I20260929 15:54:35.261096 14056 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd0745debd414fe386785de6c3c00e04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7591019df79842d48ccd6ebb72a3d605. Latest consensus state: current_term: 1 leader_uuid: "7591019df79842d48ccd6ebb72a3d605" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7591019df79842d48ccd6ebb72a3d605" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33689 } } peers { permanent_uuid: "45af35641f96452694ee520b440eb6a9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44131 } } peers { permanent_uuid: "dd0745debd414fe386785de6c3c00e04" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41517 } } }
I20260929 15:54:35.261482 14056 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd0745debd414fe386785de6c3c00e04 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:54:35.262610 14075 mvcc.cc:205] Tried to move back new op lower bound from 7334696039304081408 to 7334696039198863360. Current Snapshot: MvccSnapshot[applied={T|T < 7334696039304081408}]
I20260929 15:54:35.267051 14076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45af35641f96452694ee520b440eb6a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:54:35.276270 14056 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd0745debd414fe386785de6c3c00e04 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7591019df79842d48ccd6ebb72a3d605" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7591019df79842d48ccd6ebb72a3d605" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33689 } } peers { permanent_uuid: "45af35641f96452694ee520b440eb6a9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44131 } } peers { permanent_uuid: "dd0745debd414fe386785de6c3c00e04" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41517 } } }
I20260929 15:54:35.276522 14056 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd0745debd414fe386785de6c3c00e04 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:54:35.276845 14069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7591019df79842d48ccd6ebb72a3d605 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7591019df79842d48ccd6ebb72a3d605" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7591019df79842d48ccd6ebb72a3d605" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33689 } } peers { permanent_uuid: "45af35641f96452694ee520b440eb6a9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44131 } } peers { permanent_uuid: "dd0745debd414fe386785de6c3c00e04" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41517 } } }
I20260929 15:54:35.277132 14069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7591019df79842d48ccd6ebb72a3d605 [sys.catalog]: This master's current role is: LEADER
I20260929 15:54:35.278954 14071 catalog_manager.cc:1347] Generated new cluster ID: 37e7f27cf64744dfa99f98705ccf2786
I20260929 15:54:35.279066 14071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:54:35.326818 14076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45af35641f96452694ee520b440eb6a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7591019df79842d48ccd6ebb72a3d605. Latest consensus state: current_term: 1 leader_uuid: "7591019df79842d48ccd6ebb72a3d605" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7591019df79842d48ccd6ebb72a3d605" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33689 } } peers { permanent_uuid: "45af35641f96452694ee520b440eb6a9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44131 } } peers { permanent_uuid: "dd0745debd414fe386785de6c3c00e04" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41517 } } }
I20260929 15:54:35.327157 14076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45af35641f96452694ee520b440eb6a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:54:35.328742 14087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7591019df79842d48ccd6ebb72a3d605 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7591019df79842d48ccd6ebb72a3d605" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7591019df79842d48ccd6ebb72a3d605" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33689 } } peers { permanent_uuid: "45af35641f96452694ee520b440eb6a9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44131 } } peers { permanent_uuid: "dd0745debd414fe386785de6c3c00e04" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41517 } } }
I20260929 15:54:35.329161 14087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7591019df79842d48ccd6ebb72a3d605 [sys.catalog]: This master's current role is: LEADER
I20260929 15:54:35.331250 14085 mvcc.cc:205] Tried to move back new op lower bound from 7334696039304081408 to 7334696039198863360. Current Snapshot: MvccSnapshot[applied={T|T < 7334696039304081408}]
I20260929 15:54:35.338896 14071 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:54:35.342182 14071 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:54:35.344655 14076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45af35641f96452694ee520b440eb6a9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7591019df79842d48ccd6ebb72a3d605" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7591019df79842d48ccd6ebb72a3d605" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33689 } } peers { permanent_uuid: "45af35641f96452694ee520b440eb6a9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44131 } } peers { permanent_uuid: "dd0745debd414fe386785de6c3c00e04" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41517 } } }
I20260929 15:54:35.344888 14076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45af35641f96452694ee520b440eb6a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:54:35.358386 14094 catalog_manager.cc:1259] Loaded cluster ID: 37e7f27cf64744dfa99f98705ccf2786
I20260929 15:54:35.358551 14094 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 45af35641f96452694ee520b440eb6a9: loading cluster ID for follower catalog manager: success
I20260929 15:54:35.366942 14094 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 45af35641f96452694ee520b440eb6a9: acquiring CA information for follower catalog manager: success
I20260929 15:54:35.368880 14071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7591019df79842d48ccd6ebb72a3d605: Generated new TSK 0
I20260929 15:54:35.369772 14071 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:54:35.371526 14094 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 45af35641f96452694ee520b440eb6a9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 15:54:36.094619 14067 catalog_manager.cc:1259] Loaded cluster ID: 37e7f27cf64744dfa99f98705ccf2786
I20260929 15:54:36.094719 14067 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dd0745debd414fe386785de6c3c00e04: loading cluster ID for follower catalog manager: success
I20260929 15:54:36.097051 14067 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dd0745debd414fe386785de6c3c00e04: acquiring CA information for follower catalog manager: success
I20260929 15:54:36.098784 14067 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dd0745debd414fe386785de6c3c00e04: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 15:54:36.518898 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 13814
I20260929 15:54:36.520128 13876 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:54:36.630089 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 13814
I20260929 15:54:36.650914 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 13888
I20260929 15:54:36.651886 13950 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:54:36.757807 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 13888
I20260929 15:54:36.777741 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 13962
I20260929 15:54:36.778810 14024 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:54:36.885691 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 13962
2026-09-29T15:54:36Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4624 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:54:36 dist-test-slave-r0ck krb5kdc[14108](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:54:36 dist-test-slave-r0ck krb5kdc[14108](info): set up 2 sockets
Sep 29 15:54:36 dist-test-slave-r0ck krb5kdc[14108](info): commencing operation
krb5kdc: starting...
W20260929 15:54:37.977923 388 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:54:38 dist-test-slave-r0ck krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:54:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:54:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:54:38.201196 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42195
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42195
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:54:38.522294 14121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:54:38.522550 14121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:54:38.522720 14121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:54:38.529788 14121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:54:38.529898 14121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:54:38.530006 14121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:54:38.530084 14121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:54:38.530182 14121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:54:38.539760 14121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:54:38.540975 14121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:54:38.542232 14121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:54:38.553550 14129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:38.553597 14127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:38.553769 14126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:54:38.554665 14121 server_base.cc:1034] running on GCE node
I20260929 15:54:38.555415 14121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:54:38.556656 14121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:54:38.557891 14121 hybrid_clock.cc:648] HybridClock initialized: now 1790697278557846 us; error 46 us; skew 500 ppm
I20260929 15:54:38.558265 14121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:54:38 dist-test-slave-r0ck krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:54:38.561206 14121 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:54:38.562666 14121 webserver.cc:460] Webserver started at http://127.0.97.62:35367/ using document root <none> and password file <none>
I20260929 15:54:38.563315 14121 fs_manager.cc:359] Metadata directory not provided
I20260929 15:54:38.563442 14121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:54:38.563764 14121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:54:38.566637 14121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "84536e04a1b540f9b5c158ebae508d77"
format_stamp: "Formatted at 2026-09-29 15:54:38 on dist-test-slave-r0ck"
server_key: "d7982d22060a6ba1b37f632f8245784b"
server_key_iv: "e7ec16f872bae141a791eba9b8dce6d0"
server_key_version: "encryptionkey@0"
I20260929 15:54:38.567359 14121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "84536e04a1b540f9b5c158ebae508d77"
format_stamp: "Formatted at 2026-09-29 15:54:38 on dist-test-slave-r0ck"
server_key: "d7982d22060a6ba1b37f632f8245784b"
server_key_iv: "e7ec16f872bae141a791eba9b8dce6d0"
server_key_version: "encryptionkey@0"
I20260929 15:54:38.572190 14121 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260929 15:54:38.575526 14136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:38.576783 14121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 15:54:38.576947 14121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "84536e04a1b540f9b5c158ebae508d77"
format_stamp: "Formatted at 2026-09-29 15:54:38 on dist-test-slave-r0ck"
server_key: "d7982d22060a6ba1b37f632f8245784b"
server_key_iv: "e7ec16f872bae141a791eba9b8dce6d0"
server_key_version: "encryptionkey@0"
I20260929 15:54:38.577150 14121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:54:38.602522 14121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:54:38.606452 14121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:54:38.606758 14121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:54:38.625313 14121 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42195
I20260929 15:54:38.625387 14187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42195 every 8 connection(s)
I20260929 15:54:38.626647 14121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:54:38.630144 14188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:54:38.632186 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 14121
I20260929 15:54:38.632400 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:54:38.633015 388 external_mini_cluster.cc:1428] Setting key fdb207082c20418b99554905a86f5261
I20260929 15:54:38.641865 14188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84536e04a1b540f9b5c158ebae508d77: Bootstrap starting.
I20260929 15:54:38.645576 14188 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84536e04a1b540f9b5c158ebae508d77: Neither blocks nor log segments found. Creating new log.
I20260929 15:54:38.646886 14188 log.cc:824] T 00000000000000000000000000000000 P 84536e04a1b540f9b5c158ebae508d77: Log is configured to *not* fsync() on all Append() calls
Sep 29 15:54:38 dist-test-slave-r0ck krb5kdc[14108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:54:38.650300 14188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84536e04a1b540f9b5c158ebae508d77: No bootstrap required, opened a new log
I20260929 15:54:38.657124 14191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:38.634915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59158 (local address 127.0.97.62:42195)
0929 15:54:38.635708 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:38.635719 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:38.635758 (+ 39us) server_negotiation.cc:407] Connection header received
0929 15:54:38.637409 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:38.637437 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:38.637810 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:38.638124 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:38.639946 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:38.641255 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:38.643141 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:38.643848 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:38.643986 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:38.649524 (+ 5538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:38.649531 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:38.649546 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:38.649584 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:38.652744 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:38.653634 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:38.653641 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:38.653650 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:38.653813 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:38.654292 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:38.654296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:38.654299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:38.654777 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:38.655021 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:38.655601 (+ 580us) server_negotiation.cc:299] Negotiation successful
0929 15:54:38.656084 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":357,"threads_started":1}
I20260929 15:54:38.659363 14188 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84536e04a1b540f9b5c158ebae508d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84536e04a1b540f9b5c158ebae508d77" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42195 } }
I20260929 15:54:38.659708 14188 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84536e04a1b540f9b5c158ebae508d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:54:38.659811 14188 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84536e04a1b540f9b5c158ebae508d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84536e04a1b540f9b5c158ebae508d77, State: Initialized, Role: FOLLOWER
I20260929 15:54:38.660341 14188 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84536e04a1b540f9b5c158ebae508d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84536e04a1b540f9b5c158ebae508d77" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42195 } }
I20260929 15:54:38.660599 14188 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84536e04a1b540f9b5c158ebae508d77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:54:38.660691 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:54:38.660749 14188 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84536e04a1b540f9b5c158ebae508d77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:54:38.660894 14188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84536e04a1b540f9b5c158ebae508d77 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:54:38.663823 14188 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84536e04a1b540f9b5c158ebae508d77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84536e04a1b540f9b5c158ebae508d77" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42195 } }
I20260929 15:54:38.664374 14188 leader_election.cc:302] T 00000000000000000000000000000000 P 84536e04a1b540f9b5c158ebae508d77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84536e04a1b540f9b5c158ebae508d77; no voters:
I20260929 15:54:38.665030 14188 leader_election.cc:288] T 00000000000000000000000000000000 P 84536e04a1b540f9b5c158ebae508d77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:54:38.665184 14193 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84536e04a1b540f9b5c158ebae508d77 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:54:38.665876 14193 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84536e04a1b540f9b5c158ebae508d77 [term 1 LEADER]: Becoming Leader. State: Replica: 84536e04a1b540f9b5c158ebae508d77, State: Running, Role: LEADER
I20260929 15:54:38.666385 14193 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84536e04a1b540f9b5c158ebae508d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84536e04a1b540f9b5c158ebae508d77" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42195 } }
I20260929 15:54:38.667009 14188 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84536e04a1b540f9b5c158ebae508d77 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:54:38.669499 14194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84536e04a1b540f9b5c158ebae508d77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84536e04a1b540f9b5c158ebae508d77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84536e04a1b540f9b5c158ebae508d77" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42195 } } }
I20260929 15:54:38.669870 14194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84536e04a1b540f9b5c158ebae508d77 [sys.catalog]: This master's current role is: LEADER
I20260929 15:54:38.670230 14195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84536e04a1b540f9b5c158ebae508d77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84536e04a1b540f9b5c158ebae508d77. Latest consensus state: current_term: 1 leader_uuid: "84536e04a1b540f9b5c158ebae508d77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84536e04a1b540f9b5c158ebae508d77" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42195 } } }
I20260929 15:54:38.670584 14195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84536e04a1b540f9b5c158ebae508d77 [sys.catalog]: This master's current role is: LEADER
I20260929 15:54:38.670825 14202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:54:38.676975 14202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:54:38.685668 14202 catalog_manager.cc:1347] Generated new cluster ID: 81a2c2243a99478385bb9752e84eac92
I20260929 15:54:38.685770 14202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:54:38.716176 14202 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:54:38.717553 14202 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:54:38.727843 14202 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84536e04a1b540f9b5c158ebae508d77: Generated new TSK 0
I20260929 15:54:38.728646 14202 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:54:38.773056 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 14121
I20260929 15:54:38.774189 14183 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:54:38.869602 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 14121
2026-09-29T15:54:38Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1979 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:54:38 dist-test-slave-r0ck krb5kdc[14216](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:54:38 dist-test-slave-r0ck krb5kdc[14216](info): set up 2 sockets
Sep 29 15:54:38 dist-test-slave-r0ck krb5kdc[14216](info): commencing operation
krb5kdc: starting...
W20260929 15:54:39.977870 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.005s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:54:40 dist-test-slave-r0ck krb5kdc[14216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:54:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:54:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:54:40.201046 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45647
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:54:40.519322 14229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:54:40.519650 14229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:54:40.519811 14229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:54:40.526862 14229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:54:40.527014 14229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:54:40.527127 14229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:54:40.527228 14229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:54:40.527307 14229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:54:40.536701 14229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:54:40.538022 14229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:54:40.539288 14229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:54:40.550817 14234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:40.550773 14237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:40.550783 14235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:54:40.552007 14229 server_base.cc:1034] running on GCE node
I20260929 15:54:40.552685 14229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:54:40.553999 14229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:54:40.555227 14229 hybrid_clock.cc:648] HybridClock initialized: now 1790697280555192 us; error 44 us; skew 500 ppm
I20260929 15:54:40.555558 14229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:54:40 dist-test-slave-r0ck krb5kdc[14216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697280, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:54:40.558516 14229 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:54:40.559926 14229 webserver.cc:460] Webserver started at http://127.0.97.62:45133/ using document root <none> and password file <none>
I20260929 15:54:40.560573 14229 fs_manager.cc:359] Metadata directory not provided
I20260929 15:54:40.560730 14229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:54:40.561019 14229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:54:40.563933 14229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "b71b1df01d6748c1b6af1b4ec396e424"
format_stamp: "Formatted at 2026-09-29 15:54:40 on dist-test-slave-r0ck"
server_key: "5ad7883c77124b91a7f1c902381aabf1"
server_key_iv: "929a31de88676a3de6da37aaff8b32a6"
server_key_version: "encryptionkey@0"
I20260929 15:54:40.564685 14229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "b71b1df01d6748c1b6af1b4ec396e424"
format_stamp: "Formatted at 2026-09-29 15:54:40 on dist-test-slave-r0ck"
server_key: "5ad7883c77124b91a7f1c902381aabf1"
server_key_iv: "929a31de88676a3de6da37aaff8b32a6"
server_key_version: "encryptionkey@0"
I20260929 15:54:40.569684 14229 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 15:54:40.572927 14244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:40.574208 14229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:54:40.574383 14229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "b71b1df01d6748c1b6af1b4ec396e424"
format_stamp: "Formatted at 2026-09-29 15:54:40 on dist-test-slave-r0ck"
server_key: "5ad7883c77124b91a7f1c902381aabf1"
server_key_iv: "929a31de88676a3de6da37aaff8b32a6"
server_key_version: "encryptionkey@0"
I20260929 15:54:40.574579 14229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:54:40.613746 14229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:54:40.617877 14229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:54:40.618176 14229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:54:40.637588 14229 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45647
I20260929 15:54:40.637634 14295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45647 every 8 connection(s)
I20260929 15:54:40.638931 14229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:54:40.642084 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 14229
I20260929 15:54:40.642297 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:54:40.642735 14296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:54:40.643006 388 external_mini_cluster.cc:1428] Setting key 70fda2165d3861bb8ddbe328123081db
I20260929 15:54:40.656930 14296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424: Bootstrap starting.
Sep 29 15:54:40 dist-test-slave-r0ck krb5kdc[14216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:54:40.660732 14296 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424: Neither blocks nor log segments found. Creating new log.
I20260929 15:54:40.662141 14296 log.cc:824] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424: Log is configured to *not* fsync() on all Append() calls
I20260929 15:54:40.665663 14296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424: No bootstrap required, opened a new log
I20260929 15:54:40.667450 14299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:40.644907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43810 (local address 127.0.97.62:45647)
0929 15:54:40.645763 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:40.645778 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:40.645826 (+ 48us) server_negotiation.cc:407] Connection header received
0929 15:54:40.647581 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:40.647607 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:40.648086 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:40.648520 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:40.650051 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:40.652044 (+ 1993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:40.653486 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:40.654590 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:40.654763 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:40.658653 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:40.658664 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:40.658684 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:40.658737 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:40.662720 (+ 3983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:40.663693 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:40.663701 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:40.663709 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:40.663906 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:40.664281 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:40.664286 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:40.664289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:40.664979 (+ 690us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:40.665276 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:40.665696 (+ 420us) server_negotiation.cc:299] Negotiation successful
0929 15:54:40.666160 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":381,"threads_started":1}
I20260929 15:54:40.674664 14296 raft_consensus.cc:348] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71b1df01d6748c1b6af1b4ec396e424" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45647 } }
I20260929 15:54:40.675035 14296 raft_consensus.cc:374] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:54:40.675110 14296 raft_consensus.cc:729] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b71b1df01d6748c1b6af1b4ec396e424, State: Initialized, Role: FOLLOWER
I20260929 15:54:40.675652 14296 consensus_queue.cc:260] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71b1df01d6748c1b6af1b4ec396e424" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45647 } }
I20260929 15:54:40.675848 14296 raft_consensus.cc:388] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:54:40.675945 14296 raft_consensus.cc:482] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:54:40.676060 14296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:54:40.678315 14296 raft_consensus.cc:504] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71b1df01d6748c1b6af1b4ec396e424" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45647 } }
I20260929 15:54:40.678797 14296 leader_election.cc:302] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b71b1df01d6748c1b6af1b4ec396e424; no voters:
I20260929 15:54:40.679455 14296 leader_election.cc:288] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:54:40.679662 14301 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:54:40.680411 14301 raft_consensus.cc:686] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [term 1 LEADER]: Becoming Leader. State: Replica: b71b1df01d6748c1b6af1b4ec396e424, State: Running, Role: LEADER
I20260929 15:54:40.680984 14301 consensus_queue.cc:237] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71b1df01d6748c1b6af1b4ec396e424" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45647 } }
I20260929 15:54:40.681633 14296 sys_catalog.cc:564] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:54:40.683707 14303 sys_catalog.cc:455] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b71b1df01d6748c1b6af1b4ec396e424. Latest consensus state: current_term: 1 leader_uuid: "b71b1df01d6748c1b6af1b4ec396e424" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71b1df01d6748c1b6af1b4ec396e424" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45647 } } }
I20260929 15:54:40.683733 14302 sys_catalog.cc:455] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b71b1df01d6748c1b6af1b4ec396e424" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71b1df01d6748c1b6af1b4ec396e424" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45647 } } }
I20260929 15:54:40.684154 14303 sys_catalog.cc:458] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [sys.catalog]: This master's current role is: LEADER
I20260929 15:54:40.685274 14302 sys_catalog.cc:458] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [sys.catalog]: This master's current role is: LEADER
I20260929 15:54:40.690546 14306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:54:40.695574 14306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:54:40.703971 14306 catalog_manager.cc:1347] Generated new cluster ID: b3752e210cf14c11b1ef8533b38bb782
I20260929 15:54:40.704082 14306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:54:40.760871 14306 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:54:40.762724 14306 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:54:40.769440 14306 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424: Generated new TSK 0
I20260929 15:54:40.770236 14306 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 15:54:40.838028 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45647
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:54:41.163426 14322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:54:41.163679 14322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:54:41.163834 14322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:54:41.171644 14322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:54:41.171743 14322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:54:41.171947 14322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:54:41.181563 14322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:54:41.182775 14322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:54:41.183997 14322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:54:41.196224 14328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:41.196305 14327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:41.196275 14330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:54:41.197320 14322 server_base.cc:1034] running on GCE node
I20260929 15:54:41.197899 14322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:54:41.198750 14322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:54:41.199988 14322 hybrid_clock.cc:648] HybridClock initialized: now 1790697281199943 us; error 58 us; skew 500 ppm
I20260929 15:54:41.200311 14322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:54:41 dist-test-slave-r0ck krb5kdc[14216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:54:41.203330 14322 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:54:41.204708 14322 webserver.cc:460] Webserver started at http://127.0.97.1:34541/ using document root <none> and password file <none>
I20260929 15:54:41.205416 14322 fs_manager.cc:359] Metadata directory not provided
I20260929 15:54:41.205556 14322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:54:41.205854 14322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:54:41.208593 14322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/data/instance:
uuid: "40e6f6abaa6f4c6e8b23e1485098eb5a"
format_stamp: "Formatted at 2026-09-29 15:54:41 on dist-test-slave-r0ck"
server_key: "100e5ae7ebadf0a431ae8c4f322770d2"
server_key_iv: "c525c6b7c498ffd2ee115416e9ec929d"
server_key_version: "encryptionkey@0"
I20260929 15:54:41.209436 14322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "40e6f6abaa6f4c6e8b23e1485098eb5a"
format_stamp: "Formatted at 2026-09-29 15:54:41 on dist-test-slave-r0ck"
server_key: "100e5ae7ebadf0a431ae8c4f322770d2"
server_key_iv: "c525c6b7c498ffd2ee115416e9ec929d"
server_key_version: "encryptionkey@0"
I20260929 15:54:41.214293 14322 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 15:54:41.217694 14337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:41.218972 14322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 15:54:41.219157 14322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/wal
uuid: "40e6f6abaa6f4c6e8b23e1485098eb5a"
format_stamp: "Formatted at 2026-09-29 15:54:41 on dist-test-slave-r0ck"
server_key: "100e5ae7ebadf0a431ae8c4f322770d2"
server_key_iv: "c525c6b7c498ffd2ee115416e9ec929d"
server_key_version: "encryptionkey@0"
I20260929 15:54:41.219360 14322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:54:41.250463 14322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:54:41.254941 14322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:54:41.255239 14322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:54:41.256259 14322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:54:41.257611 14322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:54:41.257710 14322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:41.257854 14322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:54:41.257932 14322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:41.291869 14322 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36887
I20260929 15:54:41.291911 14449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36887 every 8 connection(s)
I20260929 15:54:41.293141 14322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:54:41.298735 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 14322
I20260929 15:54:41.298941 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:54:41.299574 388 external_mini_cluster.cc:1428] Setting key 3a2470cdc187da8e1b84a665180d5af8
Sep 29 15:54:41 dist-test-slave-r0ck krb5kdc[14216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:54:41.314371 14452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:41.295814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51541 (local address 127.0.97.62:45647)
0929 15:54:41.296254 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:41.296259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:41.297159 (+ 900us) server_negotiation.cc:407] Connection header received
0929 15:54:41.299060 (+ 1901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:41.299066 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:41.299158 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:41.299277 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:41.300938 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:41.301985 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:41.303499 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:41.304203 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:41.304353 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:41.308320 (+ 3967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:41.308328 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:41.308335 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:41.308374 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:41.311196 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:41.312106 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:41.312109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:41.312111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:41.312236 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:41.312709 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:41.312713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:41.312714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:41.312957 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:41.313066 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:41.313934 (+ 868us) server_negotiation.cc:299] Negotiation successful
0929 15:54:41.314095 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":214,"threads_started":1}
I20260929 15:54:41.315797 14453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:41.296321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.1:51541)
0929 15:54:41.297006 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:41.297046 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 15:54:41.298751 (+ 1705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:54:41.299503 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:54:41.299511 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:54:41.300045 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:54:41.300769 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:41.300794 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:41.302154 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:41.302158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:41.303342 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:41.303354 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:41.304411 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:41.304416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:41.304608 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:41.305285 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:54:41.305306 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:54:41.308093 (+ 2787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:54:41.311338 (+ 3245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:41.311349 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:41.311367 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:41.311997 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:41.312344 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:41.312351 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:41.312357 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:41.312596 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:41.313083 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:54:41.313093 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:54:41.313556 (+ 463us) client_negotiation.cc:769] Sending connection context
0929 15:54:41.313863 (+ 307us) client_negotiation.cc:240] Negotiation successful
0929 15:54:41.314393 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":277,"threads_started":1}
I20260929 15:54:41.317375 14450 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45647
I20260929 15:54:41.317684 14450 heartbeater.cc:464] Registering TS with master...
I20260929 15:54:41.318357 14450 heartbeater.cc:511] Master 127.0.97.62:45647 requested a full tablet report, sending...
I20260929 15:54:41.320405 14261 ts_manager.cc:194] Registered new tserver with Master: 40e6f6abaa6f4c6e8b23e1485098eb5a (127.0.97.1:36887)
I20260929 15:54:41.321897 14261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:51541
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 15:54:41.352757 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45647
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:54:41.676627 14456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:54:41.676888 14456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:54:41.676981 14456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:54:41.684588 14456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:54:41.684696 14456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:54:41.684808 14456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:54:41.694523 14456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:54:41.695703 14456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:54:41.696908 14456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:54:41.709733 14461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:41.709708 14464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:41.709793 14462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:54:41.710711 14456 server_base.cc:1034] running on GCE node
I20260929 15:54:41.711354 14456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:54:41.712234 14456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:54:41.713451 14456 hybrid_clock.cc:648] HybridClock initialized: now 1790697281713375 us; error 57 us; skew 500 ppm
I20260929 15:54:41.713778 14456 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:54:41 dist-test-slave-r0ck krb5kdc[14216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:54:41.717031 14456 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:54:41.718519 14456 webserver.cc:460] Webserver started at http://127.0.97.2:41911/ using document root <none> and password file <none>
I20260929 15:54:41.719193 14456 fs_manager.cc:359] Metadata directory not provided
I20260929 15:54:41.719340 14456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:54:41.719635 14456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:54:41.722534 14456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/data/instance:
uuid: "c58ad1c480124e0c95775e4816ec6af3"
format_stamp: "Formatted at 2026-09-29 15:54:41 on dist-test-slave-r0ck"
server_key: "73e61c8381046e8690ea9675601284d6"
server_key_iv: "8ec8c07f40200e240ddd43d5b32eb93f"
server_key_version: "encryptionkey@0"
I20260929 15:54:41.723395 14456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c58ad1c480124e0c95775e4816ec6af3"
format_stamp: "Formatted at 2026-09-29 15:54:41 on dist-test-slave-r0ck"
server_key: "73e61c8381046e8690ea9675601284d6"
server_key_iv: "8ec8c07f40200e240ddd43d5b32eb93f"
server_key_version: "encryptionkey@0"
I20260929 15:54:41.728315 14456 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260929 15:54:41.731701 14471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:41.732985 14456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 15:54:41.733165 14456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/wal
uuid: "c58ad1c480124e0c95775e4816ec6af3"
format_stamp: "Formatted at 2026-09-29 15:54:41 on dist-test-slave-r0ck"
server_key: "73e61c8381046e8690ea9675601284d6"
server_key_iv: "8ec8c07f40200e240ddd43d5b32eb93f"
server_key_version: "encryptionkey@0"
I20260929 15:54:41.733394 14456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:54:41.757884 14456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:54:41.762245 14456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:54:41.762581 14456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:54:41.763597 14456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:54:41.764850 14456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:54:41.764923 14456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:41.765062 14456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:54:41.765120 14456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:41.797608 14456 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42721
I20260929 15:54:41.797689 14583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42721 every 8 connection(s)
I20260929 15:54:41.798815 14456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:54:41.804541 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 14456
I20260929 15:54:41.804879 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:54:41.805505 388 external_mini_cluster.cc:1428] Setting key 59cc36a9ab2e44acbac0bc5f4a38aefc
Sep 29 15:54:41 dist-test-slave-r0ck krb5kdc[14216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:54:41.821642 14452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:41.801251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60301 (local address 127.0.97.62:45647)
0929 15:54:41.801519 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:41.801523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:41.802972 (+ 1449us) server_negotiation.cc:407] Connection header received
0929 15:54:41.805776 (+ 2804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:41.805782 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:41.805877 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:41.806021 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:41.808034 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:41.808779 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:41.810347 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:41.810775 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:41.810909 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:41.814776 (+ 3867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:41.814783 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:41.814790 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:41.814832 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:41.818406 (+ 3574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:41.819116 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:41.819124 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:41.819130 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:41.819282 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:41.819677 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:41.819684 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:41.819690 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:41.820059 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:41.820210 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:41.820863 (+ 653us) server_negotiation.cc:299] Negotiation successful
0929 15:54:41.821140 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 15:54:41.822311 14586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:41.801800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.2:60301)
0929 15:54:41.802770 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:41.802817 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 15:54:41.805477 (+ 2660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:54:41.806391 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:54:41.806403 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:54:41.807104 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:54:41.807874 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:41.807899 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:41.808924 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:41.808930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:41.810196 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:41.810207 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:41.810989 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:41.810995 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:41.811241 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:41.811920 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:54:41.811940 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:54:41.814554 (+ 2614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:54:41.818554 (+ 4000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:41.818560 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:41.818572 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:41.818977 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:41.819385 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:41.819388 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:41.819390 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:41.819559 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:41.820177 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:54:41.820182 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:54:41.820470 (+ 288us) client_negotiation.cc:769] Sending connection context
0929 15:54:41.820712 (+ 242us) client_negotiation.cc:240] Negotiation successful
0929 15:54:41.821080 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":376,"threads_started":1}
I20260929 15:54:41.823765 14584 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45647
I20260929 15:54:41.824080 14584 heartbeater.cc:464] Registering TS with master...
I20260929 15:54:41.824767 14584 heartbeater.cc:511] Master 127.0.97.62:45647 requested a full tablet report, sending...
I20260929 15:54:41.826308 14261 ts_manager.cc:194] Registered new tserver with Master: c58ad1c480124e0c95775e4816ec6af3 (127.0.97.2:42721)
I20260929 15:54:41.827093 14261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:60301
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 15:54:41.856734 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45647
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:54:42.179162 14589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:54:42.179468 14589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:54:42.179634 14589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:54:42.186897 14589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:54:42.187043 14589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:54:42.187222 14589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:54:42.196856 14589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:54:42.198125 14589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:54:42.199344 14589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:54:42.211984 14594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:42.212003 14597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:42.211912 14595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:54:42.212637 14589 server_base.cc:1034] running on GCE node
I20260929 15:54:42.213194 14589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:54:42.214108 14589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:54:42.215332 14589 hybrid_clock.cc:648] HybridClock initialized: now 1790697282215300 us; error 47 us; skew 500 ppm
I20260929 15:54:42.215634 14589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:54:42 dist-test-slave-r0ck krb5kdc[14216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:54:42.223179 14589 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:54:42.224758 14589 webserver.cc:460] Webserver started at http://127.0.97.3:35317/ using document root <none> and password file <none>
I20260929 15:54:42.225514 14589 fs_manager.cc:359] Metadata directory not provided
I20260929 15:54:42.225678 14589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:54:42.225992 14589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:54:42.228844 14589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/data/instance:
uuid: "4e3912abb36648b9beed2653cd515807"
format_stamp: "Formatted at 2026-09-29 15:54:42 on dist-test-slave-r0ck"
server_key: "672328c21ee8ba9890770a5c289c403c"
server_key_iv: "a5bd2d44cfa91404e0d3a320744b182c"
server_key_version: "encryptionkey@0"
I20260929 15:54:42.229704 14589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4e3912abb36648b9beed2653cd515807"
format_stamp: "Formatted at 2026-09-29 15:54:42 on dist-test-slave-r0ck"
server_key: "672328c21ee8ba9890770a5c289c403c"
server_key_iv: "a5bd2d44cfa91404e0d3a320744b182c"
server_key_version: "encryptionkey@0"
I20260929 15:54:42.234760 14589 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 15:54:42.238209 14604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:42.239394 14589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:54:42.239585 14589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/wal
uuid: "4e3912abb36648b9beed2653cd515807"
format_stamp: "Formatted at 2026-09-29 15:54:42 on dist-test-slave-r0ck"
server_key: "672328c21ee8ba9890770a5c289c403c"
server_key_iv: "a5bd2d44cfa91404e0d3a320744b182c"
server_key_version: "encryptionkey@0"
I20260929 15:54:42.239789 14589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:54:42.250917 14589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:54:42.254559 14589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:54:42.254840 14589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:54:42.255733 14589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:54:42.256955 14589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:54:42.257059 14589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:42.257212 14589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:54:42.257285 14589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:42.290477 14589 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43165
I20260929 15:54:42.290576 14716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43165 every 8 connection(s)
I20260929 15:54:42.291844 14589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:54:42.298202 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 14589
I20260929 15:54:42.298381 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:54:42.298835 388 external_mini_cluster.cc:1428] Setting key 4d0902e834c290b2ba5d207602b66a16
Sep 29 15:54:42 dist-test-slave-r0ck krb5kdc[14216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:54:42.311579 14452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:42.294525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43783 (local address 127.0.97.62:45647)
0929 15:54:42.294702 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:42.294707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:42.295884 (+ 1177us) server_negotiation.cc:407] Connection header received
0929 15:54:42.297705 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:42.297710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:42.297817 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:42.297930 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:42.299571 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:42.300727 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:42.302489 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:42.302901 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:42.302998 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:42.306285 (+ 3287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:42.306290 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:42.306293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:42.306324 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:42.308937 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:42.309664 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:42.309668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:42.309670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:42.309790 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:42.310163 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:42.310166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:42.310168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:42.310402 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:42.310513 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:42.311163 (+ 650us) server_negotiation.cc:299] Negotiation successful
0929 15:54:42.311322 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 15:54:42.312587 14719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:42.295035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.3:43783)
0929 15:54:42.295729 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:42.295768 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 15:54:42.297442 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:54:42.298156 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:54:42.298164 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:54:42.298732 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:54:42.299337 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:42.299362 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:42.300931 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:42.300935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:42.302358 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:42.302374 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:42.303077 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:42.303081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:42.303258 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:42.303963 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:54:42.303984 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:54:42.306102 (+ 2118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:54:42.309071 (+ 2969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:42.309077 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:42.309088 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:42.309558 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:42.309894 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:42.309898 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:42.309900 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:42.310065 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:42.310554 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:54:42.310560 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:54:42.310829 (+ 269us) client_negotiation.cc:769] Sending connection context
0929 15:54:42.311048 (+ 219us) client_negotiation.cc:240] Negotiation successful
0929 15:54:42.311406 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":287,"threads_started":1}
I20260929 15:54:42.313910 14717 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45647
I20260929 15:54:42.314220 14717 heartbeater.cc:464] Registering TS with master...
I20260929 15:54:42.314837 14717 heartbeater.cc:511] Master 127.0.97.62:45647 requested a full tablet report, sending...
I20260929 15:54:42.316123 14260 ts_manager.cc:194] Registered new tserver with Master: 4e3912abb36648b9beed2653cd515807 (127.0.97.3:43165)
I20260929 15:54:42.316915 14260 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:43783
I20260929 15:54:42.324921 14450 heartbeater.cc:503] Master 127.0.97.62:45647 was elected leader, sending a full tablet report...
I20260929 15:54:42.326339 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:54:42.343674 14452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:42.330717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43826 (local address 127.0.97.62:45647)
0929 15:54:42.330906 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:42.330914 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:42.331249 (+ 335us) server_negotiation.cc:407] Connection header received
0929 15:54:42.331448 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:42.331453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:42.331524 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:42.331629 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:42.333011 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:42.333819 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:42.335172 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:42.335579 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:42.335708 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:42.338220 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:42.338225 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:42.338228 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:42.338258 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:42.340863 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:42.341635 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:42.341643 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:42.341646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:42.341775 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:42.342179 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:42.342192 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:42.342194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:42.342502 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:42.342799 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:42.343124 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 15:54:42.343370 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 15:54:42.347620 14260 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:54:42.354498 14260 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:54:42.389957 14733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:42.366928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43165 (local address 127.0.0.1:57848)
0929 15:54:42.380566 (+ 13638us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:42.380590 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 15:54:42.380763 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:54:42.382012 (+ 1249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:54:42.382020 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:54:42.382055 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:54:42.382397 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:42.382405 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:42.384395 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:42.384400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:42.386753 (+ 2353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:42.386768 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:42.388861 (+ 2093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:42.388871 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:42.389069 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:42.389126 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 15:54:42.389255 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 15:54:42.389484 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13439,"mutex_wait_us":9,"thread_start_us":12230,"threads_started":1}
I20260929 15:54:42.390753 14732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:42.367347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57848 (local address 127.0.97.3:43165)
0929 15:54:42.377022 (+ 9675us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:42.377037 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:42.380928 (+ 3891us) server_negotiation.cc:407] Connection header received
0929 15:54:42.381142 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:42.381157 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:42.381508 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:42.381718 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:54:42.382578 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:42.384239 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:42.386938 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:42.388338 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:42.388498 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:42.388742 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:42.390215 (+ 1473us) server_negotiation.cc:299] Negotiation successful
0929 15:54:42.390391 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9379,"thread_start_us":9179,"threads_started":1}
I20260929 15:54:42.390793 14729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:42.366914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42721 (local address 127.0.0.1:52042)
0929 15:54:42.379693 (+ 12779us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:42.379750 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 15:54:42.380003 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:54:42.383208 (+ 3205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:54:42.383213 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:54:42.383261 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:54:42.383649 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:42.383659 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:42.385822 (+ 2163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:42.385827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:42.387873 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:42.387890 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:42.389837 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:42.389844 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:42.390126 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:42.390162 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 15:54:42.390251 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 15:54:42.390420 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12418,"mutex_wait_us":83,"thread_start_us":15044,"threads_started":1}
I20260929 15:54:42.391155 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:42.369623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43492 (local address 127.0.97.1:36887)
0929 15:54:42.377917 (+ 8294us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:42.377929 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:42.382399 (+ 4470us) server_negotiation.cc:407] Connection header received
0929 15:54:42.382635 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:42.382643 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:42.382859 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:42.383116 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:54:42.383870 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:42.385574 (+ 1704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:42.387794 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:42.389200 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:42.389364 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:42.389587 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:42.390662 (+ 1075us) server_negotiation.cc:299] Negotiation successful
0929 15:54:42.390824 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7651,"thread_start_us":7482,"threads_started":1}
I20260929 15:54:42.391701 14731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:42.367283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52042 (local address 127.0.97.2:42721)
0929 15:54:42.382686 (+ 15403us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:42.382694 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:42.382718 (+ 24us) server_negotiation.cc:407] Connection header received
0929 15:54:42.382800 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:42.382808 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:42.383003 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:42.383870 (+ 867us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:54:42.384127 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:42.385678 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:42.388056 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:42.389553 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:42.389690 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:42.389889 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:42.391215 (+ 1326us) server_negotiation.cc:299] Negotiation successful
0929 15:54:42.391439 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15238,"thread_start_us":14374,"threads_started":1}
I20260929 15:54:42.392114 14730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:42.368813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36887 (local address 127.0.0.1:43492)
0929 15:54:42.382264 (+ 13451us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:42.382279 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 15:54:42.382445 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:54:42.383140 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:54:42.383144 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:54:42.383171 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:54:42.383644 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:42.383656 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:42.385717 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:42.385723 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:42.387637 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:42.387649 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:42.389843 (+ 2194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:42.389851 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:42.390441 (+ 590us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:42.390464 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 15:54:42.390619 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 15:54:42.390744 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13312,"mutex_wait_us":114,"spinlock_wait_cycles":1408,"thread_start_us":11364,"threads_started":1}
I20260929 15:54:42.395100 14519 tablet_service.cc:1467] Processing CreateTablet for tablet 301185e6d5ed453c9fc2694b2b90fd1f (DEFAULT_TABLE table=test-table [id=c28663ff913343c6924c54292f07a28a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:54:42.395155 14652 tablet_service.cc:1467] Processing CreateTablet for tablet 301185e6d5ed453c9fc2694b2b90fd1f (DEFAULT_TABLE table=test-table [id=c28663ff913343c6924c54292f07a28a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:54:42.395413 14385 tablet_service.cc:1467] Processing CreateTablet for tablet 301185e6d5ed453c9fc2694b2b90fd1f (DEFAULT_TABLE table=test-table [id=c28663ff913343c6924c54292f07a28a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:54:42.396354 14519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 301185e6d5ed453c9fc2694b2b90fd1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:54:42.396515 14652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 301185e6d5ed453c9fc2694b2b90fd1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:54:42.396610 14385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 301185e6d5ed453c9fc2694b2b90fd1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:54:42.408362 14735 tablet_bootstrap.cc:492] T 301185e6d5ed453c9fc2694b2b90fd1f P c58ad1c480124e0c95775e4816ec6af3: Bootstrap starting.
I20260929 15:54:42.410557 14736 tablet_bootstrap.cc:492] T 301185e6d5ed453c9fc2694b2b90fd1f P 40e6f6abaa6f4c6e8b23e1485098eb5a: Bootstrap starting.
I20260929 15:54:42.411628 14737 tablet_bootstrap.cc:492] T 301185e6d5ed453c9fc2694b2b90fd1f P 4e3912abb36648b9beed2653cd515807: Bootstrap starting.
I20260929 15:54:42.413154 14735 tablet_bootstrap.cc:654] T 301185e6d5ed453c9fc2694b2b90fd1f P c58ad1c480124e0c95775e4816ec6af3: Neither blocks nor log segments found. Creating new log.
I20260929 15:54:42.413971 14736 tablet_bootstrap.cc:654] T 301185e6d5ed453c9fc2694b2b90fd1f P 40e6f6abaa6f4c6e8b23e1485098eb5a: Neither blocks nor log segments found. Creating new log.
I20260929 15:54:42.414274 14737 tablet_bootstrap.cc:654] T 301185e6d5ed453c9fc2694b2b90fd1f P 4e3912abb36648b9beed2653cd515807: Neither blocks nor log segments found. Creating new log.
I20260929 15:54:42.414464 14735 log.cc:824] T 301185e6d5ed453c9fc2694b2b90fd1f P c58ad1c480124e0c95775e4816ec6af3: Log is configured to *not* fsync() on all Append() calls
I20260929 15:54:42.415182 14736 log.cc:824] T 301185e6d5ed453c9fc2694b2b90fd1f P 40e6f6abaa6f4c6e8b23e1485098eb5a: Log is configured to *not* fsync() on all Append() calls
I20260929 15:54:42.415396 14737 log.cc:824] T 301185e6d5ed453c9fc2694b2b90fd1f P 4e3912abb36648b9beed2653cd515807: Log is configured to *not* fsync() on all Append() calls
I20260929 15:54:42.418141 14735 tablet_bootstrap.cc:492] T 301185e6d5ed453c9fc2694b2b90fd1f P c58ad1c480124e0c95775e4816ec6af3: No bootstrap required, opened a new log
I20260929 15:54:42.418293 14737 tablet_bootstrap.cc:492] T 301185e6d5ed453c9fc2694b2b90fd1f P 4e3912abb36648b9beed2653cd515807: No bootstrap required, opened a new log
I20260929 15:54:42.418416 14735 ts_tablet_manager.cc:1397] T 301185e6d5ed453c9fc2694b2b90fd1f P c58ad1c480124e0c95775e4816ec6af3: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 15:54:42.418526 14737 ts_tablet_manager.cc:1397] T 301185e6d5ed453c9fc2694b2b90fd1f P 4e3912abb36648b9beed2653cd515807: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 15:54:42.418671 14736 tablet_bootstrap.cc:492] T 301185e6d5ed453c9fc2694b2b90fd1f P 40e6f6abaa6f4c6e8b23e1485098eb5a: No bootstrap required, opened a new log
I20260929 15:54:42.418915 14736 ts_tablet_manager.cc:1397] T 301185e6d5ed453c9fc2694b2b90fd1f P 40e6f6abaa6f4c6e8b23e1485098eb5a: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 15:54:42.426404 14736 raft_consensus.cc:348] T 301185e6d5ed453c9fc2694b2b90fd1f P 40e6f6abaa6f4c6e8b23e1485098eb5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58ad1c480124e0c95775e4816ec6af3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42721 } } peers { permanent_uuid: "4e3912abb36648b9beed2653cd515807" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43165 } } peers { permanent_uuid: "40e6f6abaa6f4c6e8b23e1485098eb5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36887 } }
I20260929 15:54:42.426849 14736 raft_consensus.cc:374] T 301185e6d5ed453c9fc2694b2b90fd1f P 40e6f6abaa6f4c6e8b23e1485098eb5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:54:42.426939 14736 raft_consensus.cc:729] T 301185e6d5ed453c9fc2694b2b90fd1f P 40e6f6abaa6f4c6e8b23e1485098eb5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40e6f6abaa6f4c6e8b23e1485098eb5a, State: Initialized, Role: FOLLOWER
I20260929 15:54:42.427593 14736 consensus_queue.cc:260] T 301185e6d5ed453c9fc2694b2b90fd1f P 40e6f6abaa6f4c6e8b23e1485098eb5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58ad1c480124e0c95775e4816ec6af3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42721 } } peers { permanent_uuid: "4e3912abb36648b9beed2653cd515807" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43165 } } peers { permanent_uuid: "40e6f6abaa6f4c6e8b23e1485098eb5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36887 } }
I20260929 15:54:42.429505 14736 ts_tablet_manager.cc:1428] T 301185e6d5ed453c9fc2694b2b90fd1f P 40e6f6abaa6f4c6e8b23e1485098eb5a: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20260929 15:54:42.429219 14737 raft_consensus.cc:348] T 301185e6d5ed453c9fc2694b2b90fd1f P 4e3912abb36648b9beed2653cd515807 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58ad1c480124e0c95775e4816ec6af3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42721 } } peers { permanent_uuid: "4e3912abb36648b9beed2653cd515807" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43165 } } peers { permanent_uuid: "40e6f6abaa6f4c6e8b23e1485098eb5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36887 } }
I20260929 15:54:42.429219 14735 raft_consensus.cc:348] T 301185e6d5ed453c9fc2694b2b90fd1f P c58ad1c480124e0c95775e4816ec6af3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58ad1c480124e0c95775e4816ec6af3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42721 } } peers { permanent_uuid: "4e3912abb36648b9beed2653cd515807" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43165 } } peers { permanent_uuid: "40e6f6abaa6f4c6e8b23e1485098eb5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36887 } }
I20260929 15:54:42.429666 14737 raft_consensus.cc:374] T 301185e6d5ed453c9fc2694b2b90fd1f P 4e3912abb36648b9beed2653cd515807 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:54:42.429666 14735 raft_consensus.cc:374] T 301185e6d5ed453c9fc2694b2b90fd1f P c58ad1c480124e0c95775e4816ec6af3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:54:42.429761 14735 raft_consensus.cc:729] T 301185e6d5ed453c9fc2694b2b90fd1f P c58ad1c480124e0c95775e4816ec6af3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c58ad1c480124e0c95775e4816ec6af3, State: Initialized, Role: FOLLOWER
I20260929 15:54:42.429760 14737 raft_consensus.cc:729] T 301185e6d5ed453c9fc2694b2b90fd1f P 4e3912abb36648b9beed2653cd515807 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e3912abb36648b9beed2653cd515807, State: Initialized, Role: FOLLOWER
I20260929 15:54:42.430343 14737 consensus_queue.cc:260] T 301185e6d5ed453c9fc2694b2b90fd1f P 4e3912abb36648b9beed2653cd515807 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58ad1c480124e0c95775e4816ec6af3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42721 } } peers { permanent_uuid: "4e3912abb36648b9beed2653cd515807" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43165 } } peers { permanent_uuid: "40e6f6abaa6f4c6e8b23e1485098eb5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36887 } }
I20260929 15:54:42.430342 14735 consensus_queue.cc:260] T 301185e6d5ed453c9fc2694b2b90fd1f P c58ad1c480124e0c95775e4816ec6af3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58ad1c480124e0c95775e4816ec6af3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42721 } } peers { permanent_uuid: "4e3912abb36648b9beed2653cd515807" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43165 } } peers { permanent_uuid: "40e6f6abaa6f4c6e8b23e1485098eb5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36887 } }
I20260929 15:54:42.431871 14717 heartbeater.cc:503] Master 127.0.97.62:45647 was elected leader, sending a full tablet report...
I20260929 15:54:42.432072 14735 ts_tablet_manager.cc:1428] T 301185e6d5ed453c9fc2694b2b90fd1f P c58ad1c480124e0c95775e4816ec6af3: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260929 15:54:42.432408 14737 ts_tablet_manager.cc:1428] T 301185e6d5ed453c9fc2694b2b90fd1f P 4e3912abb36648b9beed2653cd515807: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260929 15:54:42.432574 14584 heartbeater.cc:503] Master 127.0.97.62:45647 was elected leader, sending a full tablet report...
W20260929 15:54:42.543540 14718 tablet.cc:2369] T 301185e6d5ed453c9fc2694b2b90fd1f P 4e3912abb36648b9beed2653cd515807: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:54:42.545502 14451 tablet.cc:2369] T 301185e6d5ed453c9fc2694b2b90fd1f P 40e6f6abaa6f4c6e8b23e1485098eb5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:54:42.550644 14585 tablet.cc:2369] T 301185e6d5ed453c9fc2694b2b90fd1f P c58ad1c480124e0c95775e4816ec6af3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:54:42.611210 14741 raft_consensus.cc:482] T 301185e6d5ed453c9fc2694b2b90fd1f P 40e6f6abaa6f4c6e8b23e1485098eb5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:54:42.611516 14741 raft_consensus.cc:504] T 301185e6d5ed453c9fc2694b2b90fd1f P 40e6f6abaa6f4c6e8b23e1485098eb5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58ad1c480124e0c95775e4816ec6af3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42721 } } peers { permanent_uuid: "4e3912abb36648b9beed2653cd515807" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43165 } } peers { permanent_uuid: "40e6f6abaa6f4c6e8b23e1485098eb5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36887 } }
I20260929 15:54:42.613545 14741 leader_election.cc:288] T 301185e6d5ed453c9fc2694b2b90fd1f P 40e6f6abaa6f4c6e8b23e1485098eb5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c58ad1c480124e0c95775e4816ec6af3 (127.0.97.2:42721), 4e3912abb36648b9beed2653cd515807 (127.0.97.3:43165)
I20260929 15:54:42.620249 14732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:42.614048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47703 (local address 127.0.97.3:43165)
0929 15:54:42.614258 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:42.614267 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:42.615046 (+ 779us) server_negotiation.cc:407] Connection header received
0929 15:54:42.615251 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:42.615255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:42.615313 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:42.615403 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:54:42.616042 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:42.616763 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:42.618551 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:42.619291 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:42.619440 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:42.619551 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:42.619858 (+ 307us) server_negotiation.cc:299] Negotiation successful
0929 15:54:42.619976 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 15:54:42.620285 14731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:42.613905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46877 (local address 127.0.97.2:42721)
0929 15:54:42.614113 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:42.614120 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:42.614954 (+ 834us) server_negotiation.cc:407] Connection header received
0929 15:54:42.615166 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:42.615180 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:42.615241 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:42.615336 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:54:42.615914 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:42.616624 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:42.618540 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:42.619302 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:42.619451 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:42.619556 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:42.619883 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 15:54:42.620019 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 15:54:42.620428 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:42.614086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43165 (local address 127.0.97.1:47703)
0929 15:54:42.614934 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:42.614948 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 15:54:42.615090 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:54:42.615416 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:54:42.615420 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:54:42.615446 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:54:42.615869 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:42.615878 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:42.616896 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:42.616905 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:42.618408 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:42.618419 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:42.619507 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:42.619513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:42.619685 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:42.619724 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 15:54:42.619826 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 15:54:42.619952 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":4,"thread_start_us":491,"threads_started":1}
I20260929 15:54:42.620453 14744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:42.613994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42721 (local address 127.0.97.1:46877)
0929 15:54:42.614842 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:42.614858 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:54:42.615026 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:54:42.615351 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:54:42.615355 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:54:42.615377 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:54:42.615770 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:42.615780 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:42.616747 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:42.616755 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:42.618386 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:42.618397 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:42.619545 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:42.619558 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:42.619726 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:42.619748 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 15:54:42.619848 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 15:54:42.619974 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"mutex_wait_us":99,"spinlock_wait_cycles":22400,"thread_start_us":470,"threads_started":1}
I20260929 15:54:42.621497 14672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "301185e6d5ed453c9fc2694b2b90fd1f" candidate_uuid: "40e6f6abaa6f4c6e8b23e1485098eb5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e3912abb36648b9beed2653cd515807" is_pre_election: true
I20260929 15:54:42.621531 14539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "301185e6d5ed453c9fc2694b2b90fd1f" candidate_uuid: "40e6f6abaa6f4c6e8b23e1485098eb5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c58ad1c480124e0c95775e4816ec6af3" is_pre_election: true
I20260929 15:54:42.621979 14539 raft_consensus.cc:2454] T 301185e6d5ed453c9fc2694b2b90fd1f P c58ad1c480124e0c95775e4816ec6af3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40e6f6abaa6f4c6e8b23e1485098eb5a in term 0.
I20260929 15:54:42.621994 14672 raft_consensus.cc:2454] T 301185e6d5ed453c9fc2694b2b90fd1f P 4e3912abb36648b9beed2653cd515807 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40e6f6abaa6f4c6e8b23e1485098eb5a in term 0.
I20260929 15:54:42.622661 14339 leader_election.cc:302] T 301185e6d5ed453c9fc2694b2b90fd1f P 40e6f6abaa6f4c6e8b23e1485098eb5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 40e6f6abaa6f4c6e8b23e1485098eb5a, 4e3912abb36648b9beed2653cd515807, c58ad1c480124e0c95775e4816ec6af3; no voters:
I20260929 15:54:42.623077 14741 raft_consensus.cc:2781] T 301185e6d5ed453c9fc2694b2b90fd1f P 40e6f6abaa6f4c6e8b23e1485098eb5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:54:42.623243 14741 raft_consensus.cc:482] T 301185e6d5ed453c9fc2694b2b90fd1f P 40e6f6abaa6f4c6e8b23e1485098eb5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:54:42.623355 14741 raft_consensus.cc:3037] T 301185e6d5ed453c9fc2694b2b90fd1f P 40e6f6abaa6f4c6e8b23e1485098eb5a [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:54:42.625664 14741 raft_consensus.cc:504] T 301185e6d5ed453c9fc2694b2b90fd1f P 40e6f6abaa6f4c6e8b23e1485098eb5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58ad1c480124e0c95775e4816ec6af3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42721 } } peers { permanent_uuid: "4e3912abb36648b9beed2653cd515807" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43165 } } peers { permanent_uuid: "40e6f6abaa6f4c6e8b23e1485098eb5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36887 } }
I20260929 15:54:42.626204 14741 leader_election.cc:288] T 301185e6d5ed453c9fc2694b2b90fd1f P 40e6f6abaa6f4c6e8b23e1485098eb5a [CANDIDATE]: Term 1 election: Requested vote from peers c58ad1c480124e0c95775e4816ec6af3 (127.0.97.2:42721), 4e3912abb36648b9beed2653cd515807 (127.0.97.3:43165)
I20260929 15:54:42.626582 14539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "301185e6d5ed453c9fc2694b2b90fd1f" candidate_uuid: "40e6f6abaa6f4c6e8b23e1485098eb5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c58ad1c480124e0c95775e4816ec6af3"
I20260929 15:54:42.626652 14672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "301185e6d5ed453c9fc2694b2b90fd1f" candidate_uuid: "40e6f6abaa6f4c6e8b23e1485098eb5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e3912abb36648b9beed2653cd515807"
I20260929 15:54:42.626796 14539 raft_consensus.cc:3037] T 301185e6d5ed453c9fc2694b2b90fd1f P c58ad1c480124e0c95775e4816ec6af3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:54:42.626881 14672 raft_consensus.cc:3037] T 301185e6d5ed453c9fc2694b2b90fd1f P 4e3912abb36648b9beed2653cd515807 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:54:42.629057 14672 raft_consensus.cc:2454] T 301185e6d5ed453c9fc2694b2b90fd1f P 4e3912abb36648b9beed2653cd515807 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40e6f6abaa6f4c6e8b23e1485098eb5a in term 1.
I20260929 15:54:42.629307 14539 raft_consensus.cc:2454] T 301185e6d5ed453c9fc2694b2b90fd1f P c58ad1c480124e0c95775e4816ec6af3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40e6f6abaa6f4c6e8b23e1485098eb5a in term 1.
I20260929 15:54:42.629477 14339 leader_election.cc:302] T 301185e6d5ed453c9fc2694b2b90fd1f P 40e6f6abaa6f4c6e8b23e1485098eb5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40e6f6abaa6f4c6e8b23e1485098eb5a, 4e3912abb36648b9beed2653cd515807; no voters:
I20260929 15:54:42.629773 14741 raft_consensus.cc:2781] T 301185e6d5ed453c9fc2694b2b90fd1f P 40e6f6abaa6f4c6e8b23e1485098eb5a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:54:42.630179 14741 raft_consensus.cc:686] T 301185e6d5ed453c9fc2694b2b90fd1f P 40e6f6abaa6f4c6e8b23e1485098eb5a [term 1 LEADER]: Becoming Leader. State: Replica: 40e6f6abaa6f4c6e8b23e1485098eb5a, State: Running, Role: LEADER
I20260929 15:54:42.630832 14741 consensus_queue.cc:237] T 301185e6d5ed453c9fc2694b2b90fd1f P 40e6f6abaa6f4c6e8b23e1485098eb5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58ad1c480124e0c95775e4816ec6af3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42721 } } peers { permanent_uuid: "4e3912abb36648b9beed2653cd515807" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43165 } } peers { permanent_uuid: "40e6f6abaa6f4c6e8b23e1485098eb5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36887 } }
I20260929 15:54:42.635491 14261 catalog_manager.cc:5495] T 301185e6d5ed453c9fc2694b2b90fd1f P 40e6f6abaa6f4c6e8b23e1485098eb5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 40e6f6abaa6f4c6e8b23e1485098eb5a (127.0.97.1). New cstate: current_term: 1 leader_uuid: "40e6f6abaa6f4c6e8b23e1485098eb5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58ad1c480124e0c95775e4816ec6af3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e3912abb36648b9beed2653cd515807" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40e6f6abaa6f4c6e8b23e1485098eb5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36887 } health_report { overall_health: HEALTHY } } }
I20260929 15:54:42.682446 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:42.676990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43498 (local address 127.0.97.1:36887)
0929 15:54:42.677229 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:42.677233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:42.677252 (+ 19us) server_negotiation.cc:407] Connection header received
0929 15:54:42.677392 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:42.677399 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:42.677491 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:42.677642 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:54:42.678065 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:42.678865 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:42.680384 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:42.680791 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:42.680897 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:42.681328 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:54:42.681932 (+ 604us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:54:42.682050 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:42.682181 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 15:54:42.682250 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 15:54:42.690363 14672 raft_consensus.cc:1261] T 301185e6d5ed453c9fc2694b2b90fd1f P 4e3912abb36648b9beed2653cd515807 [term 1 FOLLOWER]: Refusing update from remote peer 40e6f6abaa6f4c6e8b23e1485098eb5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:54:42.690382 14539 raft_consensus.cc:1261] T 301185e6d5ed453c9fc2694b2b90fd1f P c58ad1c480124e0c95775e4816ec6af3 [term 1 FOLLOWER]: Refusing update from remote peer 40e6f6abaa6f4c6e8b23e1485098eb5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:54:42.691380 14741 consensus_queue.cc:1035] T 301185e6d5ed453c9fc2694b2b90fd1f P 40e6f6abaa6f4c6e8b23e1485098eb5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c58ad1c480124e0c95775e4816ec6af3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:54:42.691670 14746 consensus_queue.cc:1035] T 301185e6d5ed453c9fc2694b2b90fd1f P 40e6f6abaa6f4c6e8b23e1485098eb5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e3912abb36648b9beed2653cd515807" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:54:42.702913 14750 mvcc.cc:205] Tried to move back new op lower bound from 7334696069889847296 to 7334696069660545024. Current Snapshot: MvccSnapshot[applied={T|T < 7334696069889847296}]
I20260929 15:54:42.706136 14751 mvcc.cc:205] Tried to move back new op lower bound from 7334696069889847296 to 7334696069660545024. Current Snapshot: MvccSnapshot[applied={T|T < 7334696069889847296}]
I20260929 15:54:42.715461 14259 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:54:42.715767 14259 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:54:42.719357 14259 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424: Sending DeleteTablet for 3 replicas of tablet 301185e6d5ed453c9fc2694b2b90fd1f
I20260929 15:54:42.720525 14519 tablet_service.cc:1514] Processing DeleteTablet for tablet 301185e6d5ed453c9fc2694b2b90fd1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:54:42 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52042
I20260929 15:54:42.720607 14652 tablet_service.cc:1514] Processing DeleteTablet for tablet 301185e6d5ed453c9fc2694b2b90fd1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:54:42 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57848
I20260929 15:54:42.720840 14385 tablet_service.cc:1514] Processing DeleteTablet for tablet 301185e6d5ed453c9fc2694b2b90fd1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:54:42 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43492
I20260929 15:54:42.721926 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 14322
I20260929 15:54:42.727967 14759 tablet_replica.cc:330] T 301185e6d5ed453c9fc2694b2b90fd1f P 4e3912abb36648b9beed2653cd515807: stopping tablet replica
I20260929 15:54:42.728896 14760 tablet_replica.cc:330] T 301185e6d5ed453c9fc2694b2b90fd1f P 40e6f6abaa6f4c6e8b23e1485098eb5a: stopping tablet replica
I20260929 15:54:42.729302 14760 raft_consensus.cc:2229] T 301185e6d5ed453c9fc2694b2b90fd1f P 40e6f6abaa6f4c6e8b23e1485098eb5a [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:54:42.730050 14760 raft_consensus.cc:2258] T 301185e6d5ed453c9fc2694b2b90fd1f P 40e6f6abaa6f4c6e8b23e1485098eb5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:54:42.728458 14759 raft_consensus.cc:2229] T 301185e6d5ed453c9fc2694b2b90fd1f P 4e3912abb36648b9beed2653cd515807 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:54:42.730899 14758 tablet_replica.cc:330] T 301185e6d5ed453c9fc2694b2b90fd1f P c58ad1c480124e0c95775e4816ec6af3: stopping tablet replica
I20260929 15:54:42.731228 14759 raft_consensus.cc:2258] T 301185e6d5ed453c9fc2694b2b90fd1f P 4e3912abb36648b9beed2653cd515807 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:54:42.731390 14758 raft_consensus.cc:2229] T 301185e6d5ed453c9fc2694b2b90fd1f P c58ad1c480124e0c95775e4816ec6af3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:54:42.731900 14758 raft_consensus.cc:2258] T 301185e6d5ed453c9fc2694b2b90fd1f P c58ad1c480124e0c95775e4816ec6af3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:54:42.732304 14760 ts_tablet_manager.cc:1905] T 301185e6d5ed453c9fc2694b2b90fd1f P 40e6f6abaa6f4c6e8b23e1485098eb5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:54:42.733114 14759 ts_tablet_manager.cc:1905] T 301185e6d5ed453c9fc2694b2b90fd1f P 4e3912abb36648b9beed2653cd515807: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:54:42.734859 14758 ts_tablet_manager.cc:1905] T 301185e6d5ed453c9fc2694b2b90fd1f P c58ad1c480124e0c95775e4816ec6af3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:54:42.739851 14759 ts_tablet_manager.cc:1918] T 301185e6d5ed453c9fc2694b2b90fd1f P 4e3912abb36648b9beed2653cd515807: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:54:42.740015 14759 log.cc:1196] T 301185e6d5ed453c9fc2694b2b90fd1f P 4e3912abb36648b9beed2653cd515807: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/wal/wals/301185e6d5ed453c9fc2694b2b90fd1f
I20260929 15:54:42.740646 14759 ts_tablet_manager.cc:1939] T 301185e6d5ed453c9fc2694b2b90fd1f P 4e3912abb36648b9beed2653cd515807: Deleting consensus metadata
I20260929 15:54:42.740693 14760 ts_tablet_manager.cc:1918] T 301185e6d5ed453c9fc2694b2b90fd1f P 40e6f6abaa6f4c6e8b23e1485098eb5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:54:42.740907 14760 log.cc:1196] T 301185e6d5ed453c9fc2694b2b90fd1f P 40e6f6abaa6f4c6e8b23e1485098eb5a: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/wal/wals/301185e6d5ed453c9fc2694b2b90fd1f
I20260929 15:54:42.741474 14760 ts_tablet_manager.cc:1939] T 301185e6d5ed453c9fc2694b2b90fd1f P 40e6f6abaa6f4c6e8b23e1485098eb5a: Deleting consensus metadata
I20260929 15:54:42.742611 14246 catalog_manager.cc:4841] TS 4e3912abb36648b9beed2653cd515807 (127.0.97.3:43165): tablet 301185e6d5ed453c9fc2694b2b90fd1f (table test-table [id=c28663ff913343c6924c54292f07a28a]) successfully deleted
I20260929 15:54:42.743064 14245 catalog_manager.cc:4841] TS 40e6f6abaa6f4c6e8b23e1485098eb5a (127.0.97.1:36887): tablet 301185e6d5ed453c9fc2694b2b90fd1f (table test-table [id=c28663ff913343c6924c54292f07a28a]) successfully deleted
Sep 29 15:54:42 dist-test-slave-r0ck krb5kdc[14216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 15:54:42.745631 14758 ts_tablet_manager.cc:1918] T 301185e6d5ed453c9fc2694b2b90fd1f P c58ad1c480124e0c95775e4816ec6af3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:54:42.745785 14758 log.cc:1196] T 301185e6d5ed453c9fc2694b2b90fd1f P c58ad1c480124e0c95775e4816ec6af3: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/wal/wals/301185e6d5ed453c9fc2694b2b90fd1f
I20260929 15:54:42.746277 14758 ts_tablet_manager.cc:1939] T 301185e6d5ed453c9fc2694b2b90fd1f P c58ad1c480124e0c95775e4816ec6af3: Deleting consensus metadata
I20260929 15:54:42.747728 14247 catalog_manager.cc:4841] TS c58ad1c480124e0c95775e4816ec6af3 (127.0.97.2:42721): tablet 301185e6d5ed453c9fc2694b2b90fd1f (table test-table [id=c28663ff913343c6924c54292f07a28a]) successfully deleted
I20260929 15:54:42.752643 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:42.723392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43502 (local address 127.0.97.1:36887)
0929 15:54:42.723623 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:42.723648 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:42.728232 (+ 4584us) server_negotiation.cc:407] Connection header received
0929 15:54:42.728440 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:42.728448 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:42.728556 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:42.728730 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:42.731065 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:42.732137 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:42.736938 (+ 4801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:42.737701 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:42.737801 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:42.745883 (+ 8082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:42.745896 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:42.745914 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:42.745989 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:42.749919 (+ 3930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:42.750575 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:42.750584 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:42.750587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:42.750764 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:42.751097 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:42.751103 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:42.751106 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:42.751517 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:42.751818 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:42.752047 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 15:54:42.752277 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 15:54:42.753366 14445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:54:42.850845 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 14322
I20260929 15:54:42.870265 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 14456
Sep 29 15:54:42 dist-test-slave-r0ck krb5kdc[14216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 15:54:42.885591 14731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:42.871002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52058 (local address 127.0.97.2:42721)
0929 15:54:42.871198 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:42.871206 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:42.871231 (+ 25us) server_negotiation.cc:407] Connection header received
0929 15:54:42.871443 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:42.871450 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:42.871545 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:42.871677 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:42.872937 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:42.873875 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:42.875158 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:42.875549 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:42.875696 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:42.879989 (+ 4293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:42.880004 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:42.880019 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:42.880053 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:42.882827 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:42.883493 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:42.883501 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:42.883503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:42.883631 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:42.883994 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:42.884000 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:42.884003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:42.884383 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:42.884696 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:42.885010 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 15:54:42.885221 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 15:54:42.886247 14579 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:54:42.976259 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 14456
I20260929 15:54:42.993646 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 14589
Sep 29 15:54:42 dist-test-slave-r0ck krb5kdc[14216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 15:54:43.009503 14732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:42.994445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57850 (local address 127.0.97.3:43165)
0929 15:54:42.994643 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:42.994648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:42.994667 (+ 19us) server_negotiation.cc:407] Connection header received
0929 15:54:42.994822 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:42.994826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:42.994894 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:42.994982 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:42.996284 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:42.997037 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:42.998380 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:42.998759 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:42.998853 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:43.003398 (+ 4545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:43.003415 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:43.003432 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:43.003475 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:43.006396 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:43.007158 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:43.007165 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:43.007168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:43.007346 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:43.007847 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:43.007854 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:43.007857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:43.008228 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:43.008497 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:43.008817 (+ 320us) server_negotiation.cc:299] Negotiation successful
0929 15:54:43.008988 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 15:54:43.010260 14712 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:54:43.100771 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 14589
I20260929 15:54:43.117172 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 14229
I20260929 15:54:43.118312 14291 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:54:43.230651 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 14229
I20260929 15:54:43.253108 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45647
--webserver_interface=127.0.97.62
--webserver_port=45133
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:54:43.596865 14766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:54:43.597188 14766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:54:43.597375 14766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:54:43.604589 14766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:54:43.604733 14766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:54:43.604835 14766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:54:43.604934 14766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:54:43.605010 14766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:54:43.614821 14766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=45133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:54:43.616086 14766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:54:43.617331 14766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:54:43.639698 14772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:43.639675 14773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:43.639647 14775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:54:43.640594 14766 server_base.cc:1034] running on GCE node
I20260929 15:54:43.641556 14766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:54:43.643136 14766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:54:43.644381 14766 hybrid_clock.cc:648] HybridClock initialized: now 1790697283644317 us; error 52 us; skew 500 ppm
I20260929 15:54:43.644773 14766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:54:43 dist-test-slave-r0ck krb5kdc[14216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:54:43.648381 14766 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:54:43.650106 14766 webserver.cc:460] Webserver started at http://127.0.97.62:45133/ using document root <none> and password file <none>
I20260929 15:54:43.650815 14766 fs_manager.cc:359] Metadata directory not provided
I20260929 15:54:43.650926 14766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:54:43.656813 14766 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 15:54:43.659664 14782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:43.661002 14766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 15:54:43.661146 14766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "b71b1df01d6748c1b6af1b4ec396e424"
format_stamp: "Formatted at 2026-09-29 15:54:40 on dist-test-slave-r0ck"
server_key: "5ad7883c77124b91a7f1c902381aabf1"
server_key_iv: "929a31de88676a3de6da37aaff8b32a6"
server_key_version: "encryptionkey@0"
I20260929 15:54:43.662256 14766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:54:43.682215 14766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:54:43.686887 14766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:54:43.687137 14766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:54:43.706653 14766 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45647
I20260929 15:54:43.706760 14833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45647 every 8 connection(s)
I20260929 15:54:43.708158 14766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:54:43.709645 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 14766
I20260929 15:54:43.710649 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:36887
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:45647
--webserver_port=34541
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:54:43.718314 14834 sys_catalog.cc:263] Verifying existing consensus state
I20260929 15:54:43.721702 14834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424: Bootstrap starting.
I20260929 15:54:43.741814 14834 log.cc:824] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424: Log is configured to *not* fsync() on all Append() calls
I20260929 15:54:43.753551 14834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 15:54:43.754072 14834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424: Bootstrap complete.
I20260929 15:54:43.763227 14834 raft_consensus.cc:348] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71b1df01d6748c1b6af1b4ec396e424" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45647 } }
I20260929 15:54:43.764149 14834 raft_consensus.cc:729] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b71b1df01d6748c1b6af1b4ec396e424, State: Initialized, Role: FOLLOWER
I20260929 15:54:43.764796 14834 consensus_queue.cc:260] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71b1df01d6748c1b6af1b4ec396e424" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45647 } }
I20260929 15:54:43.765043 14834 raft_consensus.cc:388] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:54:43.765180 14834 raft_consensus.cc:482] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:54:43.765398 14834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [term 1 FOLLOWER]: Advancing to term 2
I20260929 15:54:43.769594 14834 raft_consensus.cc:504] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71b1df01d6748c1b6af1b4ec396e424" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45647 } }
I20260929 15:54:43.770095 14834 leader_election.cc:302] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b71b1df01d6748c1b6af1b4ec396e424; no voters:
I20260929 15:54:43.770717 14834 leader_election.cc:288] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 15:54:43.770857 14838 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 15:54:43.773639 14834 sys_catalog.cc:564] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:54:43.773707 14838 raft_consensus.cc:686] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [term 2 LEADER]: Becoming Leader. State: Replica: b71b1df01d6748c1b6af1b4ec396e424, State: Running, Role: LEADER
I20260929 15:54:43.774326 14838 consensus_queue.cc:237] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71b1df01d6748c1b6af1b4ec396e424" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45647 } }
I20260929 15:54:43.777319 14840 sys_catalog.cc:455] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b71b1df01d6748c1b6af1b4ec396e424. Latest consensus state: current_term: 2 leader_uuid: "b71b1df01d6748c1b6af1b4ec396e424" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71b1df01d6748c1b6af1b4ec396e424" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45647 } } }
I20260929 15:54:43.777719 14840 sys_catalog.cc:458] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [sys.catalog]: This master's current role is: LEADER
I20260929 15:54:43.777786 14839 sys_catalog.cc:455] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b71b1df01d6748c1b6af1b4ec396e424" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71b1df01d6748c1b6af1b4ec396e424" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45647 } } }
I20260929 15:54:43.778077 14839 sys_catalog.cc:458] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [sys.catalog]: This master's current role is: LEADER
I20260929 15:54:43.789649 14848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:54:43.797530 14848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:54:43.798987 14848 catalog_manager.cc:1259] Loaded cluster ID: b3752e210cf14c11b1ef8533b38bb782
I20260929 15:54:43.799131 14848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:54:43.812542 14848 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 15:54:43.831130 14848 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 15:54:43.841235 14848 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:54:43.842913 14848 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424: Loaded TSK: 0
I20260929 15:54:43.844471 14848 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 15:54:44.076452 14836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:54:44.076717 14836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:54:44.076858 14836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:54:44.084082 14836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:54:44.084187 14836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:54:44.084358 14836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:54:44.094352 14836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:36887
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=34541
--tserver_master_addrs=127.0.97.62:45647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:54:44.095690 14836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:54:44.096971 14836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:54:44.111753 14862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:44.111724 14861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:54:44.112429 14836 server_base.cc:1034] running on GCE node
W20260929 15:54:44.112507 14864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:54:44.113282 14836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:54:44.114454 14836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:54:44.115711 14836 hybrid_clock.cc:648] HybridClock initialized: now 1790697284115635 us; error 87 us; skew 500 ppm
I20260929 15:54:44.116079 14836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:54:44 dist-test-slave-r0ck krb5kdc[14216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:54:44.120012 14836 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:54:44.121802 14836 webserver.cc:460] Webserver started at http://127.0.97.1:34541/ using document root <none> and password file <none>
I20260929 15:54:44.122709 14836 fs_manager.cc:359] Metadata directory not provided
I20260929 15:54:44.122872 14836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:54:44.129985 14836 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 15:54:44.133213 14871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:44.134807 14836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:54:44.134986 14836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/wal
uuid: "40e6f6abaa6f4c6e8b23e1485098eb5a"
format_stamp: "Formatted at 2026-09-29 15:54:41 on dist-test-slave-r0ck"
server_key: "100e5ae7ebadf0a431ae8c4f322770d2"
server_key_iv: "c525c6b7c498ffd2ee115416e9ec929d"
server_key_version: "encryptionkey@0"
I20260929 15:54:44.136296 14836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:54:44.153923 14836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:54:44.158736 14836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:54:44.159044 14836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:54:44.160259 14836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:54:44.161983 14836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:54:44.162107 14836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:44.162211 14836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:54:44.162289 14836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:44.202133 14836 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36887
I20260929 15:54:44.202239 14983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36887 every 8 connection(s)
I20260929 15:54:44.203843 14836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:54:44.207720 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 14836
I20260929 15:54:44.208670 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:42721
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:45647
--webserver_port=41911
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 15:54:44 dist-test-slave-r0ck krb5kdc[14216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:54:44.233227 14987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:44.206903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.1:50831)
0929 15:54:44.210968 (+ 4065us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:44.211023 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 15:54:44.213759 (+ 2736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:54:44.215087 (+ 1328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:54:44.215099 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:54:44.215859 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:54:44.216625 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:44.216645 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:44.219052 (+ 2407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:44.219055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:44.220215 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:44.220227 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:44.221658 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:44.221663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:44.221841 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:44.222532 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:54:44.222553 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:54:44.224581 (+ 2028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:54:44.229458 (+ 4877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:44.229464 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:44.229474 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:44.229873 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:44.230370 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:44.230373 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:44.230375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:44.230545 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:44.231484 (+ 939us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:54:44.231489 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:54:44.231754 (+ 265us) client_negotiation.cc:769] Sending connection context
0929 15:54:44.231963 (+ 209us) client_negotiation.cc:240] Negotiation successful
0929 15:54:44.232310 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3767,"thread_start_us":3567,"threads_started":1}
I20260929 15:54:44.233974 14986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:44.206664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50831 (local address 127.0.97.62:45647)
0929 15:54:44.210652 (+ 3988us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:44.210669 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:44.213113 (+ 2444us) server_negotiation.cc:407] Connection header received
0929 15:54:44.214281 (+ 1168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:44.214294 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:44.214598 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:44.214883 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:44.216859 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:44.218908 (+ 2049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:44.220366 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:44.221457 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:44.221630 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:44.224852 (+ 3222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:44.224864 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:44.224884 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:44.224931 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:44.229298 (+ 4367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:44.230014 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:44.230025 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:44.230035 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:44.230250 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:44.230664 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:44.230671 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:44.230676 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:44.231346 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:44.231553 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:44.232231 (+ 678us) server_negotiation.cc:299] Negotiation successful
0929 15:54:44.232696 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3715,"thread_start_us":3550,"threads_started":1}
I20260929 15:54:44.236703 14984 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45647
I20260929 15:54:44.236997 14984 heartbeater.cc:464] Registering TS with master...
I20260929 15:54:44.237740 14984 heartbeater.cc:511] Master 127.0.97.62:45647 requested a full tablet report, sending...
I20260929 15:54:44.239765 14799 ts_manager.cc:194] Registered new tserver with Master: 40e6f6abaa6f4c6e8b23e1485098eb5a (127.0.97.1:36887)
I20260929 15:54:44.241295 14799 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50831
W20260929 15:54:44.531347 14988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:54:44.531605 14988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:54:44.531775 14988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:54:44.538802 14988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:54:44.538905 14988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:54:44.539099 14988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:54:44.548538 14988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:42721
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=41911
--tserver_master_addrs=127.0.97.62:45647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:54:44.549764 14988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:54:44.551096 14988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:54:44.563426 14993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:44.563495 14996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:44.563289 14994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:54:44.564390 14988 server_base.cc:1034] running on GCE node
I20260929 15:54:44.564957 14988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:54:44.566012 14988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:54:44.567241 14988 hybrid_clock.cc:648] HybridClock initialized: now 1790697284567180 us; error 53 us; skew 500 ppm
I20260929 15:54:44.567575 14988 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:54:44 dist-test-slave-r0ck krb5kdc[14216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:54:44.570659 14988 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:54:44.572048 14988 webserver.cc:460] Webserver started at http://127.0.97.2:41911/ using document root <none> and password file <none>
I20260929 15:54:44.572759 14988 fs_manager.cc:359] Metadata directory not provided
I20260929 15:54:44.572902 14988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:54:44.577904 14988 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:54:44.580375 15003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:44.581658 14988 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:54:44.581830 14988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/wal
uuid: "c58ad1c480124e0c95775e4816ec6af3"
format_stamp: "Formatted at 2026-09-29 15:54:41 on dist-test-slave-r0ck"
server_key: "73e61c8381046e8690ea9675601284d6"
server_key_iv: "8ec8c07f40200e240ddd43d5b32eb93f"
server_key_version: "encryptionkey@0"
I20260929 15:54:44.582829 14988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:54:44.610515 14988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:54:44.614348 14988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:54:44.614629 14988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:54:44.615592 14988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:54:44.617007 14988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:54:44.617152 14988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:44.617290 14988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:54:44.617398 14988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:44.650074 14988 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42721
I20260929 15:54:44.650161 15115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42721 every 8 connection(s)
I20260929 15:54:44.651623 14988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:54:44.653975 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 14988
I20260929 15:54:44.654724 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:43165
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:45647
--webserver_port=35317
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 15:54:44 dist-test-slave-r0ck krb5kdc[14216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:54:44.672238 14986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:44.654241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52935 (local address 127.0.97.62:45647)
0929 15:54:44.654415 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:44.654420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:44.655685 (+ 1265us) server_negotiation.cc:407] Connection header received
0929 15:54:44.657499 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:44.657505 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:44.657600 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:44.657715 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:44.659093 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:44.659796 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:44.661299 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:44.661868 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:44.662000 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:44.666480 (+ 4480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:44.666485 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:44.666489 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:44.666520 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:44.669093 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:44.670006 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:44.670010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:44.670012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:44.670134 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:44.670621 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:44.670624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:44.670626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:44.670872 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:44.670983 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:44.671839 (+ 856us) server_negotiation.cc:299] Negotiation successful
0929 15:54:44.672001 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 15:54:44.673840 15118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:44.654788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.2:52935)
0929 15:54:44.655513 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:44.655553 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 15:54:44.657203 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:54:44.657941 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:54:44.657948 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:54:44.658414 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:54:44.658925 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:44.658940 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:44.659966 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:44.659970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:44.661137 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:44.661148 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:44.662090 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:44.662096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:44.662419 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:44.663224 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:54:44.663257 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:54:44.666214 (+ 2957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:54:44.669231 (+ 3017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:44.669241 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:44.669258 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:44.669895 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:44.670245 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:44.670253 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:44.670260 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:44.670515 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:44.671009 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:54:44.671019 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:54:44.671427 (+ 408us) client_negotiation.cc:769] Sending connection context
0929 15:54:44.671742 (+ 315us) client_negotiation.cc:240] Negotiation successful
0929 15:54:44.672298 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":296,"threads_started":1}
I20260929 15:54:44.675441 15116 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45647
I20260929 15:54:44.675777 15116 heartbeater.cc:464] Registering TS with master...
I20260929 15:54:44.676659 15116 heartbeater.cc:511] Master 127.0.97.62:45647 requested a full tablet report, sending...
I20260929 15:54:44.678367 14799 ts_manager.cc:194] Registered new tserver with Master: c58ad1c480124e0c95775e4816ec6af3 (127.0.97.2:42721)
I20260929 15:54:44.679677 14799 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:52935
W20260929 15:54:44.975422 15119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:54:44.975674 15119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:54:44.975765 15119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:54:44.982686 15119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:54:44.982786 15119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:54:44.982893 15119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:54:44.992465 15119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:43165
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=35317
--tserver_master_addrs=127.0.97.62:45647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:54:44.993686 15119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:54:44.994876 15119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:54:45.007779 15127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:45.007779 15125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:45.008042 15124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:54:45.008602 15119 server_base.cc:1034] running on GCE node
I20260929 15:54:45.009200 15119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:54:45.010183 15119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:54:45.011448 15119 hybrid_clock.cc:648] HybridClock initialized: now 1790697285011396 us; error 35 us; skew 500 ppm
I20260929 15:54:45.011866 15119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:54:45 dist-test-slave-r0ck krb5kdc[14216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697285, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:54:45.015157 15119 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:54:45.016664 15119 webserver.cc:460] Webserver started at http://127.0.97.3:35317/ using document root <none> and password file <none>
I20260929 15:54:45.017401 15119 fs_manager.cc:359] Metadata directory not provided
I20260929 15:54:45.017565 15119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:54:45.022740 15119 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260929 15:54:45.025281 15134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:45.026559 15119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 15:54:45.026734 15119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/wal
uuid: "4e3912abb36648b9beed2653cd515807"
format_stamp: "Formatted at 2026-09-29 15:54:42 on dist-test-slave-r0ck"
server_key: "672328c21ee8ba9890770a5c289c403c"
server_key_iv: "a5bd2d44cfa91404e0d3a320744b182c"
server_key_version: "encryptionkey@0"
I20260929 15:54:45.027660 15119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:54:45.041852 15119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:54:45.045918 15119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:54:45.046188 15119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:54:45.047088 15119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:54:45.048301 15119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:54:45.048431 15119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:45.048568 15119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:54:45.048655 15119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:45.083132 15119 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43165
I20260929 15:54:45.083216 15246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43165 every 8 connection(s)
I20260929 15:54:45.084556 15119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:54:45.088560 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 15119
Sep 29 15:54:45 dist-test-slave-r0ck krb5kdc[14216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697285, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:54:45.104516 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:45.089461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43842 (local address 127.0.97.62:45647)
0929 15:54:45.090862 (+ 1401us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:45.090866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:45.090881 (+ 15us) server_negotiation.cc:407] Connection header received
0929 15:54:45.090919 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:45.090922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:45.091000 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:45.091114 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:45.092538 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:45.093296 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:45.094683 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:45.095158 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:45.095292 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:45.098216 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:45.098221 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:45.098224 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:45.098256 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:45.101663 (+ 3407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:45.102344 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:45.102349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:45.102353 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:45.102494 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:45.102916 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:45.102921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:45.102923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:45.103286 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:45.103468 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:45.103836 (+ 368us) server_negotiation.cc:299] Negotiation successful
0929 15:54:45.104047 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":1320,"spinlock_wait_cycles":11520,"thread_start_us":1156,"threads_started":1}
I20260929 15:54:45.104970 14986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:45.087050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48965 (local address 127.0.97.62:45647)
0929 15:54:45.087217 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:45.087222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:45.088239 (+ 1017us) server_negotiation.cc:407] Connection header received
0929 15:54:45.090729 (+ 2490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:45.090735 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:45.090825 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:45.090932 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:45.092352 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:45.093117 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:45.094707 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:45.095198 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:45.095331 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:45.098919 (+ 3588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:45.098924 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:45.098927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:45.098956 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:45.101771 (+ 2815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:45.102506 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:45.102510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:45.102513 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:45.102702 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:45.103213 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:45.103216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:45.103218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:45.103567 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:45.103725 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:45.104569 (+ 844us) server_negotiation.cc:299] Negotiation successful
0929 15:54:45.104727 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":50,"spinlock_wait_cycles":31488}
I20260929 15:54:45.106282 15249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:45.087433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.3:48965)
0929 15:54:45.088088 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:45.088123 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 15:54:45.090458 (+ 2335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:54:45.091163 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:54:45.091171 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:54:45.091673 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:54:45.092188 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:45.092205 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:45.093280 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:45.093285 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:45.094542 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:45.094554 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:45.095415 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:45.095422 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:45.095667 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:45.096343 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:54:45.096363 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:54:45.098702 (+ 2339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:54:45.101904 (+ 3202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:45.101911 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:45.101923 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:45.102378 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:45.102814 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:45.102817 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:45.102819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:45.103077 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:45.103696 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:54:45.103724 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:54:45.104139 (+ 415us) client_negotiation.cc:769] Sending connection context
0929 15:54:45.104453 (+ 314us) client_negotiation.cc:240] Negotiation successful
0929 15:54:45.105023 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":254,"threads_started":1}
I20260929 15:54:45.107646 15247 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45647
I20260929 15:54:45.107914 15247 heartbeater.cc:464] Registering TS with master...
I20260929 15:54:45.108557 15247 heartbeater.cc:511] Master 127.0.97.62:45647 requested a full tablet report, sending...
I20260929 15:54:45.109889 14798 ts_manager.cc:194] Registered new tserver with Master: 4e3912abb36648b9beed2653cd515807 (127.0.97.3:43165)
I20260929 15:54:45.110730 14798 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48965
I20260929 15:54:45.119401 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:54:45.125844 14986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:45.120236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43856 (local address 127.0.97.62:45647)
0929 15:54:45.120391 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:45.120395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:45.120838 (+ 443us) server_negotiation.cc:407] Connection header received
0929 15:54:45.121003 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:45.121007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:45.121142 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:45.121264 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:54:45.121680 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:45.122471 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:45.123969 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:45.124368 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:45.124464 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:45.124906 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:54:45.125410 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:54:45.125521 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:45.125611 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 15:54:45.125672 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 15:54:45.126633 14798 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:43856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:54:45.133948 14798 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:54:45.155437 15257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:45.145051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42721 (local address 127.0.0.1:52064)
0929 15:54:45.145933 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:45.145988 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 15:54:45.146205 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:54:45.147414 (+ 1209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:54:45.147421 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:54:45.147450 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:54:45.148267 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:45.148281 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:45.150474 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:45.150481 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:45.152900 (+ 2419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:45.152911 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:45.154307 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:45.154313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:45.154587 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:45.154671 (+ 84us) client_negotiation.cc:769] Sending connection context
0929 15:54:45.154811 (+ 140us) client_negotiation.cc:240] Negotiation successful
0929 15:54:45.154990 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"mutex_wait_us":54,"thread_start_us":408,"threads_started":1}
I20260929 15:54:45.156040 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:45.145444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52064 (local address 127.0.97.2:42721)
0929 15:54:45.146772 (+ 1328us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:45.146780 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:45.146811 (+ 31us) server_negotiation.cc:407] Connection header received
0929 15:54:45.146902 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:45.146909 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:45.147157 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:45.148013 (+ 856us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:54:45.148869 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:45.150283 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:45.153046 (+ 2763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:45.154082 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:45.154245 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:45.155438 (+ 1193us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:45.155517 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 15:54:45.155700 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1171,"thread_start_us":1057,"threads_started":1}
I20260929 15:54:45.157367 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:45.147425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43165 (local address 127.0.0.1:57854)
0929 15:54:45.148947 (+ 1522us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:45.148969 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:54:45.149100 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:54:45.150237 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:54:45.150241 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:54:45.150264 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:54:45.150690 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:45.150699 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:45.152284 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:45.152291 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:45.154670 (+ 2379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:45.154685 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:45.156509 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:45.156515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:45.156671 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:45.156703 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 15:54:45.156800 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 15:54:45.156933 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1352,"mutex_wait_us":67,"thread_start_us":1175,"threads_started":1}
I20260929 15:54:45.157680 15258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:45.145329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36887 (local address 127.0.0.1:43516)
0929 15:54:45.148078 (+ 2749us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:45.148097 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:54:45.148865 (+ 768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:54:45.149648 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:54:45.149652 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:54:45.149681 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:54:45.149988 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:45.149999 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:45.152655 (+ 2656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:45.152660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:45.154779 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:45.154789 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:45.156810 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:45.156818 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:45.157023 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:45.157057 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 15:54:45.157149 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 15:54:45.157294 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2583,"thread_start_us":2412,"threads_started":1}
I20260929 15:54:45.157821 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:45.145862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43516 (local address 127.0.97.1:36887)
0929 15:54:45.147585 (+ 1723us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:45.147594 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:45.148321 (+ 727us) server_negotiation.cc:407] Connection header received
0929 15:54:45.149079 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:45.149086 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:45.149293 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:45.149495 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:54:45.150180 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:45.152517 (+ 2337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:45.154958 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:45.156356 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:45.156466 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:45.156632 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:45.157270 (+ 638us) server_negotiation.cc:299] Negotiation successful
0929 15:54:45.157483 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1563,"thread_start_us":1452,"threads_started":1}
I20260929 15:54:45.158279 15262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:45.148147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57854 (local address 127.0.97.3:43165)
0929 15:54:45.149689 (+ 1542us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:45.149697 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:45.149729 (+ 32us) server_negotiation.cc:407] Connection header received
0929 15:54:45.149826 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:45.149834 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:45.150062 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:45.150243 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:54:45.150860 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:45.152157 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:45.154864 (+ 2707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:45.156291 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:45.156452 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:45.156672 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:45.156878 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 15:54:45.157037 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1405,"thread_start_us":1256,"threads_started":1}
I20260929 15:54:45.160213 15051 tablet_service.cc:1467] Processing CreateTablet for tablet 049feb5ebe714e37b137ae053dc53abd (DEFAULT_TABLE table=test-table [id=5d9ec770a4d64862b31c5983c333c9cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:54:45.161809 15051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 049feb5ebe714e37b137ae053dc53abd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:54:45.161310 14919 tablet_service.cc:1467] Processing CreateTablet for tablet 049feb5ebe714e37b137ae053dc53abd (DEFAULT_TABLE table=test-table [id=5d9ec770a4d64862b31c5983c333c9cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:54:45.161500 15182 tablet_service.cc:1467] Processing CreateTablet for tablet 049feb5ebe714e37b137ae053dc53abd (DEFAULT_TABLE table=test-table [id=5d9ec770a4d64862b31c5983c333c9cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:54:45.162535 15182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 049feb5ebe714e37b137ae053dc53abd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:54:45.162590 14919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 049feb5ebe714e37b137ae053dc53abd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:54:45.174324 15263 tablet_bootstrap.cc:492] T 049feb5ebe714e37b137ae053dc53abd P 4e3912abb36648b9beed2653cd515807: Bootstrap starting.
I20260929 15:54:45.177747 15263 tablet_bootstrap.cc:654] T 049feb5ebe714e37b137ae053dc53abd P 4e3912abb36648b9beed2653cd515807: Neither blocks nor log segments found. Creating new log.
I20260929 15:54:45.178083 15265 tablet_bootstrap.cc:492] T 049feb5ebe714e37b137ae053dc53abd P c58ad1c480124e0c95775e4816ec6af3: Bootstrap starting.
I20260929 15:54:45.179105 15263 log.cc:824] T 049feb5ebe714e37b137ae053dc53abd P 4e3912abb36648b9beed2653cd515807: Log is configured to *not* fsync() on all Append() calls
I20260929 15:54:45.180469 15264 tablet_bootstrap.cc:492] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a: Bootstrap starting.
I20260929 15:54:45.181437 15265 tablet_bootstrap.cc:654] T 049feb5ebe714e37b137ae053dc53abd P c58ad1c480124e0c95775e4816ec6af3: Neither blocks nor log segments found. Creating new log.
I20260929 15:54:45.182179 15263 tablet_bootstrap.cc:492] T 049feb5ebe714e37b137ae053dc53abd P 4e3912abb36648b9beed2653cd515807: No bootstrap required, opened a new log
I20260929 15:54:45.182439 15263 ts_tablet_manager.cc:1397] T 049feb5ebe714e37b137ae053dc53abd P 4e3912abb36648b9beed2653cd515807: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20260929 15:54:45.182669 15265 log.cc:824] T 049feb5ebe714e37b137ae053dc53abd P c58ad1c480124e0c95775e4816ec6af3: Log is configured to *not* fsync() on all Append() calls
I20260929 15:54:45.183785 15264 tablet_bootstrap.cc:654] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a: Neither blocks nor log segments found. Creating new log.
I20260929 15:54:45.184680 15264 log.cc:824] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a: Log is configured to *not* fsync() on all Append() calls
I20260929 15:54:45.185545 15265 tablet_bootstrap.cc:492] T 049feb5ebe714e37b137ae053dc53abd P c58ad1c480124e0c95775e4816ec6af3: No bootstrap required, opened a new log
I20260929 15:54:45.185838 15265 ts_tablet_manager.cc:1397] T 049feb5ebe714e37b137ae053dc53abd P c58ad1c480124e0c95775e4816ec6af3: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 15:54:45.188330 15264 tablet_bootstrap.cc:492] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a: No bootstrap required, opened a new log
I20260929 15:54:45.188591 15264 ts_tablet_manager.cc:1397] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260929 15:54:45.193096 15263 raft_consensus.cc:348] T 049feb5ebe714e37b137ae053dc53abd P 4e3912abb36648b9beed2653cd515807 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58ad1c480124e0c95775e4816ec6af3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42721 } } peers { permanent_uuid: "40e6f6abaa6f4c6e8b23e1485098eb5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36887 } } peers { permanent_uuid: "4e3912abb36648b9beed2653cd515807" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43165 } }
I20260929 15:54:45.193552 15263 raft_consensus.cc:374] T 049feb5ebe714e37b137ae053dc53abd P 4e3912abb36648b9beed2653cd515807 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:54:45.193655 15263 raft_consensus.cc:729] T 049feb5ebe714e37b137ae053dc53abd P 4e3912abb36648b9beed2653cd515807 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e3912abb36648b9beed2653cd515807, State: Initialized, Role: FOLLOWER
I20260929 15:54:45.194286 15263 consensus_queue.cc:260] T 049feb5ebe714e37b137ae053dc53abd P 4e3912abb36648b9beed2653cd515807 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58ad1c480124e0c95775e4816ec6af3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42721 } } peers { permanent_uuid: "40e6f6abaa6f4c6e8b23e1485098eb5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36887 } } peers { permanent_uuid: "4e3912abb36648b9beed2653cd515807" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43165 } }
I20260929 15:54:45.195420 15247 heartbeater.cc:503] Master 127.0.97.62:45647 was elected leader, sending a full tablet report...
I20260929 15:54:45.196090 15263 ts_tablet_manager.cc:1428] T 049feb5ebe714e37b137ae053dc53abd P 4e3912abb36648b9beed2653cd515807: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260929 15:54:45.196408 15265 raft_consensus.cc:348] T 049feb5ebe714e37b137ae053dc53abd P c58ad1c480124e0c95775e4816ec6af3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58ad1c480124e0c95775e4816ec6af3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42721 } } peers { permanent_uuid: "40e6f6abaa6f4c6e8b23e1485098eb5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36887 } } peers { permanent_uuid: "4e3912abb36648b9beed2653cd515807" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43165 } }
I20260929 15:54:45.196801 15265 raft_consensus.cc:374] T 049feb5ebe714e37b137ae053dc53abd P c58ad1c480124e0c95775e4816ec6af3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:54:45.196931 15265 raft_consensus.cc:729] T 049feb5ebe714e37b137ae053dc53abd P c58ad1c480124e0c95775e4816ec6af3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c58ad1c480124e0c95775e4816ec6af3, State: Initialized, Role: FOLLOWER
I20260929 15:54:45.196722 15264 raft_consensus.cc:348] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58ad1c480124e0c95775e4816ec6af3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42721 } } peers { permanent_uuid: "40e6f6abaa6f4c6e8b23e1485098eb5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36887 } } peers { permanent_uuid: "4e3912abb36648b9beed2653cd515807" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43165 } }
I20260929 15:54:45.197161 15264 raft_consensus.cc:374] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:54:45.197285 15264 raft_consensus.cc:729] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40e6f6abaa6f4c6e8b23e1485098eb5a, State: Initialized, Role: FOLLOWER
I20260929 15:54:45.197593 15265 consensus_queue.cc:260] T 049feb5ebe714e37b137ae053dc53abd P c58ad1c480124e0c95775e4816ec6af3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58ad1c480124e0c95775e4816ec6af3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42721 } } peers { permanent_uuid: "40e6f6abaa6f4c6e8b23e1485098eb5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36887 } } peers { permanent_uuid: "4e3912abb36648b9beed2653cd515807" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43165 } }
I20260929 15:54:45.197913 15264 consensus_queue.cc:260] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58ad1c480124e0c95775e4816ec6af3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42721 } } peers { permanent_uuid: "40e6f6abaa6f4c6e8b23e1485098eb5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36887 } } peers { permanent_uuid: "4e3912abb36648b9beed2653cd515807" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43165 } }
I20260929 15:54:45.198715 15116 heartbeater.cc:503] Master 127.0.97.62:45647 was elected leader, sending a full tablet report...
I20260929 15:54:45.199301 15265 ts_tablet_manager.cc:1428] T 049feb5ebe714e37b137ae053dc53abd P c58ad1c480124e0c95775e4816ec6af3: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260929 15:54:45.201524 14984 heartbeater.cc:503] Master 127.0.97.62:45647 was elected leader, sending a full tablet report...
I20260929 15:54:45.201996 15264 ts_tablet_manager.cc:1428] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a: Time spent starting tablet: real 0.013s user 0.012s sys 0.001s
W20260929 15:54:45.206013 14985 tablet.cc:2369] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:54:45.269794 15269 raft_consensus.cc:482] T 049feb5ebe714e37b137ae053dc53abd P 4e3912abb36648b9beed2653cd515807 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:54:45.270154 15269 raft_consensus.cc:504] T 049feb5ebe714e37b137ae053dc53abd P 4e3912abb36648b9beed2653cd515807 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58ad1c480124e0c95775e4816ec6af3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42721 } } peers { permanent_uuid: "40e6f6abaa6f4c6e8b23e1485098eb5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36887 } } peers { permanent_uuid: "4e3912abb36648b9beed2653cd515807" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43165 } }
I20260929 15:54:45.271921 15269 leader_election.cc:288] T 049feb5ebe714e37b137ae053dc53abd P 4e3912abb36648b9beed2653cd515807 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c58ad1c480124e0c95775e4816ec6af3 (127.0.97.2:42721), 40e6f6abaa6f4c6e8b23e1485098eb5a (127.0.97.1:36887)
I20260929 15:54:45.279258 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:45.272473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45645 (local address 127.0.97.2:42721)
0929 15:54:45.272670 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:45.272675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:45.272694 (+ 19us) server_negotiation.cc:407] Connection header received
0929 15:54:45.272954 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:45.272961 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:45.273041 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:45.273147 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:54:45.273610 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:45.274655 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:45.277177 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:45.278384 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:45.278516 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:45.278623 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:45.278910 (+ 287us) server_negotiation.cc:299] Negotiation successful
0929 15:54:45.279005 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 15:54:45.279326 15249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:45.272230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42721 (local address 127.0.97.3:45645)
0929 15:54:45.272584 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:45.272605 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:54:45.272724 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:54:45.273186 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:54:45.273191 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:54:45.273227 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:54:45.273472 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:45.273487 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:45.274810 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:45.274814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:45.277017 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:45.277031 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:45.278559 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:45.278563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:45.278726 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:45.278768 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 15:54:45.278876 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 15:54:45.279012 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":86}
I20260929 15:54:45.279573 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:45.272574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38219 (local address 127.0.97.1:36887)
0929 15:54:45.272803 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:45.272811 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:45.273404 (+ 593us) server_negotiation.cc:407] Connection header received
0929 15:54:45.273577 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:45.273584 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:45.273687 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:45.273780 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:54:45.274208 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:45.275073 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:45.277419 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:45.278624 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:45.278743 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:45.278804 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:45.279172 (+ 368us) server_negotiation.cc:299] Negotiation successful
0929 15:54:45.279303 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 15:54:45.279826 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:45.272230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36887 (local address 127.0.97.3:38219)
0929 15:54:45.273236 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:45.273252 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:54:45.273425 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:54:45.273792 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:54:45.273795 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:54:45.273812 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:54:45.274059 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:45.274066 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:45.275200 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:45.275205 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:45.277253 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:45.277262 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:45.278826 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:45.278834 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:45.279005 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:45.279034 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 15:54:45.279116 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 15:54:45.279232 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":878,"mutex_wait_us":344,"thread_start_us":337,"threads_started":1}
I20260929 15:54:45.280346 15071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "049feb5ebe714e37b137ae053dc53abd" candidate_uuid: "4e3912abb36648b9beed2653cd515807" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c58ad1c480124e0c95775e4816ec6af3" is_pre_election: true
I20260929 15:54:45.280818 15071 raft_consensus.cc:2454] T 049feb5ebe714e37b137ae053dc53abd P c58ad1c480124e0c95775e4816ec6af3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e3912abb36648b9beed2653cd515807 in term 0.
I20260929 15:54:45.280761 14939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "049feb5ebe714e37b137ae053dc53abd" candidate_uuid: "4e3912abb36648b9beed2653cd515807" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40e6f6abaa6f4c6e8b23e1485098eb5a" is_pre_election: true
I20260929 15:54:45.281224 14939 raft_consensus.cc:2454] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e3912abb36648b9beed2653cd515807 in term 0.
I20260929 15:54:45.281629 15137 leader_election.cc:302] T 049feb5ebe714e37b137ae053dc53abd P 4e3912abb36648b9beed2653cd515807 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e3912abb36648b9beed2653cd515807, c58ad1c480124e0c95775e4816ec6af3; no voters:
I20260929 15:54:45.282028 15269 raft_consensus.cc:2781] T 049feb5ebe714e37b137ae053dc53abd P 4e3912abb36648b9beed2653cd515807 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:54:45.282176 15269 raft_consensus.cc:482] T 049feb5ebe714e37b137ae053dc53abd P 4e3912abb36648b9beed2653cd515807 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:54:45.282289 15269 raft_consensus.cc:3037] T 049feb5ebe714e37b137ae053dc53abd P 4e3912abb36648b9beed2653cd515807 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:54:45.284730 15269 raft_consensus.cc:504] T 049feb5ebe714e37b137ae053dc53abd P 4e3912abb36648b9beed2653cd515807 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58ad1c480124e0c95775e4816ec6af3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42721 } } peers { permanent_uuid: "40e6f6abaa6f4c6e8b23e1485098eb5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36887 } } peers { permanent_uuid: "4e3912abb36648b9beed2653cd515807" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43165 } }
I20260929 15:54:45.285422 15269 leader_election.cc:288] T 049feb5ebe714e37b137ae053dc53abd P 4e3912abb36648b9beed2653cd515807 [CANDIDATE]: Term 1 election: Requested vote from peers c58ad1c480124e0c95775e4816ec6af3 (127.0.97.2:42721), 40e6f6abaa6f4c6e8b23e1485098eb5a (127.0.97.1:36887)
I20260929 15:54:45.285784 15071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "049feb5ebe714e37b137ae053dc53abd" candidate_uuid: "4e3912abb36648b9beed2653cd515807" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c58ad1c480124e0c95775e4816ec6af3"
I20260929 15:54:45.285940 14939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "049feb5ebe714e37b137ae053dc53abd" candidate_uuid: "4e3912abb36648b9beed2653cd515807" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40e6f6abaa6f4c6e8b23e1485098eb5a"
I20260929 15:54:45.285995 15071 raft_consensus.cc:3037] T 049feb5ebe714e37b137ae053dc53abd P c58ad1c480124e0c95775e4816ec6af3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:54:45.286156 14939 raft_consensus.cc:3037] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:54:45.288384 14939 raft_consensus.cc:2454] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e3912abb36648b9beed2653cd515807 in term 1.
I20260929 15:54:45.288394 15071 raft_consensus.cc:2454] T 049feb5ebe714e37b137ae053dc53abd P c58ad1c480124e0c95775e4816ec6af3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e3912abb36648b9beed2653cd515807 in term 1.
I20260929 15:54:45.288803 15135 leader_election.cc:302] T 049feb5ebe714e37b137ae053dc53abd P 4e3912abb36648b9beed2653cd515807 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40e6f6abaa6f4c6e8b23e1485098eb5a, 4e3912abb36648b9beed2653cd515807; no voters:
I20260929 15:54:45.289150 15269 raft_consensus.cc:2781] T 049feb5ebe714e37b137ae053dc53abd P 4e3912abb36648b9beed2653cd515807 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:54:45.289670 15269 raft_consensus.cc:686] T 049feb5ebe714e37b137ae053dc53abd P 4e3912abb36648b9beed2653cd515807 [term 1 LEADER]: Becoming Leader. State: Replica: 4e3912abb36648b9beed2653cd515807, State: Running, Role: LEADER
I20260929 15:54:45.290205 15269 consensus_queue.cc:237] T 049feb5ebe714e37b137ae053dc53abd P 4e3912abb36648b9beed2653cd515807 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58ad1c480124e0c95775e4816ec6af3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42721 } } peers { permanent_uuid: "40e6f6abaa6f4c6e8b23e1485098eb5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36887 } } peers { permanent_uuid: "4e3912abb36648b9beed2653cd515807" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43165 } }
I20260929 15:54:45.294888 14798 catalog_manager.cc:5495] T 049feb5ebe714e37b137ae053dc53abd P 4e3912abb36648b9beed2653cd515807 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e3912abb36648b9beed2653cd515807 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "4e3912abb36648b9beed2653cd515807" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58ad1c480124e0c95775e4816ec6af3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40e6f6abaa6f4c6e8b23e1485098eb5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e3912abb36648b9beed2653cd515807" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43165 } health_report { overall_health: HEALTHY } } }
I20260929 15:54:45.313643 15262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:45.308647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57860 (local address 127.0.97.3:43165)
0929 15:54:45.308821 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:45.308825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:45.308842 (+ 17us) server_negotiation.cc:407] Connection header received
0929 15:54:45.308961 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:45.308964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:45.309035 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:45.309150 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:54:45.309485 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:45.310205 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:45.311644 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:45.312015 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:45.312105 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:45.312524 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:54:45.312992 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:54:45.313094 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:45.313271 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 15:54:45.313406 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 15:54:45.321156 14939 raft_consensus.cc:1261] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [term 1 FOLLOWER]: Refusing update from remote peer 4e3912abb36648b9beed2653cd515807: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:54:45.321182 15071 raft_consensus.cc:1261] T 049feb5ebe714e37b137ae053dc53abd P c58ad1c480124e0c95775e4816ec6af3 [term 1 FOLLOWER]: Refusing update from remote peer 4e3912abb36648b9beed2653cd515807: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:54:45.322121 15273 consensus_queue.cc:1035] T 049feb5ebe714e37b137ae053dc53abd P 4e3912abb36648b9beed2653cd515807 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c58ad1c480124e0c95775e4816ec6af3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:54:45.322441 15269 consensus_queue.cc:1035] T 049feb5ebe714e37b137ae053dc53abd P 4e3912abb36648b9beed2653cd515807 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40e6f6abaa6f4c6e8b23e1485098eb5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:54:45.348407 14798 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:43856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:54:45.348703 14798 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:43856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:54:45.352232 14798 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424: Sending DeleteTablet for 3 replicas of tablet 049feb5ebe714e37b137ae053dc53abd
I20260929 15:54:45.353483 15182 tablet_service.cc:1514] Processing DeleteTablet for tablet 049feb5ebe714e37b137ae053dc53abd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:54:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57854
I20260929 15:54:45.353518 14919 tablet_service.cc:1514] Processing DeleteTablet for tablet 049feb5ebe714e37b137ae053dc53abd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:54:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43516
I20260929 15:54:45.353547 15051 tablet_service.cc:1514] Processing DeleteTablet for tablet 049feb5ebe714e37b137ae053dc53abd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:54:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52064
I20260929 15:54:45.354115 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 14836
I20260929 15:54:45.358131 15286 tablet_replica.cc:330] T 049feb5ebe714e37b137ae053dc53abd P 4e3912abb36648b9beed2653cd515807: stopping tablet replica
I20260929 15:54:45.358606 15286 raft_consensus.cc:2229] T 049feb5ebe714e37b137ae053dc53abd P 4e3912abb36648b9beed2653cd515807 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:54:45.360836 15286 raft_consensus.cc:2258] T 049feb5ebe714e37b137ae053dc53abd P 4e3912abb36648b9beed2653cd515807 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:54:45.363682 15286 ts_tablet_manager.cc:1905] T 049feb5ebe714e37b137ae053dc53abd P 4e3912abb36648b9beed2653cd515807: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:54:45.365866 15288 tablet_replica.cc:330] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a: stopping tablet replica
I20260929 15:54:45.366865 15288 raft_consensus.cc:2229] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:54:45.367913 15288 raft_consensus.cc:2258] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:54:45.370847 15287 tablet_replica.cc:330] T 049feb5ebe714e37b137ae053dc53abd P c58ad1c480124e0c95775e4816ec6af3: stopping tablet replica
I20260929 15:54:45.371346 15287 raft_consensus.cc:2229] T 049feb5ebe714e37b137ae053dc53abd P c58ad1c480124e0c95775e4816ec6af3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:54:45.371899 15287 raft_consensus.cc:2258] T 049feb5ebe714e37b137ae053dc53abd P c58ad1c480124e0c95775e4816ec6af3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:54:45.371974 15286 ts_tablet_manager.cc:1918] T 049feb5ebe714e37b137ae053dc53abd P 4e3912abb36648b9beed2653cd515807: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:54:45.372220 15286 log.cc:1196] T 049feb5ebe714e37b137ae053dc53abd P 4e3912abb36648b9beed2653cd515807: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/wal/wals/049feb5ebe714e37b137ae053dc53abd
I20260929 15:54:45.373181 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:45.354980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43532 (local address 127.0.97.1:36887)
0929 15:54:45.355156 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:45.355165 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:45.355293 (+ 128us) server_negotiation.cc:407] Connection header received
0929 15:54:45.355494 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:45.355500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:45.355582 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:45.355688 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:45.357791 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:45.358996 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:45.360499 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:45.361096 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:45.361217 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:45.364721 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:45.364736 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:45.364752 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:45.364789 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:45.368935 (+ 4146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:45.369852 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:45.369860 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:45.369864 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:45.370019 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:45.370481 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:45.370485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:45.370488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:45.370945 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:45.371220 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:45.372555 (+ 1335us) server_negotiation.cc:299] Negotiation successful
0929 15:54:45.372786 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 15:54:45.373390 15286 ts_tablet_manager.cc:1939] T 049feb5ebe714e37b137ae053dc53abd P 4e3912abb36648b9beed2653cd515807: Deleting consensus metadata
I20260929 15:54:45.374097 14979 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:54:45.375473 14784 catalog_manager.cc:4841] TS 4e3912abb36648b9beed2653cd515807 (127.0.97.3:43165): tablet 049feb5ebe714e37b137ae053dc53abd (table test-table [id=5d9ec770a4d64862b31c5983c333c9cb]) successfully deleted
I20260929 15:54:45.376160 15287 ts_tablet_manager.cc:1905] T 049feb5ebe714e37b137ae053dc53abd P c58ad1c480124e0c95775e4816ec6af3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:54:45.383266 15287 ts_tablet_manager.cc:1918] T 049feb5ebe714e37b137ae053dc53abd P c58ad1c480124e0c95775e4816ec6af3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:54:45.383423 15287 log.cc:1196] T 049feb5ebe714e37b137ae053dc53abd P c58ad1c480124e0c95775e4816ec6af3: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/wal/wals/049feb5ebe714e37b137ae053dc53abd
I20260929 15:54:45.384123 15287 ts_tablet_manager.cc:1939] T 049feb5ebe714e37b137ae053dc53abd P c58ad1c480124e0c95775e4816ec6af3: Deleting consensus metadata
I20260929 15:54:45.386132 14785 catalog_manager.cc:4841] TS c58ad1c480124e0c95775e4816ec6af3 (127.0.97.2:42721): tablet 049feb5ebe714e37b137ae053dc53abd (table test-table [id=5d9ec770a4d64862b31c5983c333c9cb]) successfully deleted
I20260929 15:54:45.466995 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 14836
I20260929 15:54:45.485109 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 14988
W20260929 15:54:45.485188 14783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:36887 (error 108)
I20260929 15:54:45.487174 15258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:45.486593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36887 (local address 127.0.0.1:43544)
0929 15:54:45.486842 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:45.486953 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
W20260929 15:54:45.487432 14783 catalog_manager.cc:4584] TS 40e6f6abaa6f4c6e8b23e1485098eb5a (127.0.97.1:36887): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 049feb5ebe714e37b137ae053dc53abd: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36887: connect: Connection refused (error 111)
I20260929 15:54:45.499608 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:45.485906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52070 (local address 127.0.97.2:42721)
0929 15:54:45.486092 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:45.486098 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:45.486410 (+ 312us) server_negotiation.cc:407] Connection header received
0929 15:54:45.486483 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:45.486487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:45.486548 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:45.486660 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:45.488190 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:45.488933 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:45.490558 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:45.490965 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:45.491102 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:45.493823 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:45.493837 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:45.493850 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:45.493882 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:45.496759 (+ 2877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:45.497565 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:45.497573 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:45.497575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:45.497701 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:45.498100 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:45.498104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:45.498106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:45.498460 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:45.498771 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:45.499054 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 15:54:45.499272 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 15:54:45.500670 15111 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:54:45.541128 15258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:45.540573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36887 (local address 127.0.0.1:43558)
0929 15:54:45.540822 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:45.540932 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260929 15:54:45.542480 15258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:45.542040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36887 (local address 127.0.0.1:43560)
0929 15:54:45.542223 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:45.542312 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 15:54:45.594511 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 14988
I20260929 15:54:45.603318 15258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:45.602691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36887 (local address 127.0.0.1:43572)
0929 15:54:45.603006 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:45.603111 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260929 15:54:45.604544 15258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:45.604107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36887 (local address 127.0.0.1:43584)
0929 15:54:45.604294 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:45.604395 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 15:54:45.612869 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 15119
I20260929 15:54:45.632043 15262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:45.613707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57862 (local address 127.0.97.3:43165)
0929 15:54:45.613867 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:45.613871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:45.613889 (+ 18us) server_negotiation.cc:407] Connection header received
0929 15:54:45.614099 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:45.614103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:45.614179 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:45.614280 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:45.616109 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:45.616926 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:45.619029 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:45.619489 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:45.619593 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:45.623986 (+ 4393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:45.624011 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:45.624033 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:45.624099 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:45.628370 (+ 4271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:45.629542 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:45.629550 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:45.629553 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:45.629742 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:45.630217 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:45.630221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:45.630223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:45.630688 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:45.631049 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:45.631344 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 15:54:45.631573 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 15:54:45.632915 15242 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:54:45.685389 15258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:45.684805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36887 (local address 127.0.0.1:43600)
0929 15:54:45.685045 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:45.685171 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260929 15:54:45.686537 15258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:45.686098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36887 (local address 127.0.0.1:43612)
0929 15:54:45.686325 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:45.686407 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260929 15:54:45.756052 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 15119
I20260929 15:54:45.771739 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 14766
I20260929 15:54:45.772951 14829 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:54:45.890182 15258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:45.889664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36887 (local address 127.0.0.1:43626)
0929 15:54:45.889840 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:45.889936 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260929 15:54:45.890179 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 14766
I20260929 15:54:45.911972 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45647
--webserver_interface=127.0.97.62
--webserver_port=45133
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:54:46.243520 15294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:54:46.243777 15294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:54:46.243929 15294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:54:46.251061 15294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:54:46.251183 15294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:54:46.251266 15294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:54:46.251365 15294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:54:46.251431 15294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:54:46.261675 15294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=45133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:54:46.262933 15294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:54:46.264190 15294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:54:46.275606 15300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:46.275861 15299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:46.275666 15302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:54:46.276551 15294 server_base.cc:1034] running on GCE node
I20260929 15:54:46.277272 15294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:54:46.278604 15294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:54:46.279800 15294 hybrid_clock.cc:648] HybridClock initialized: now 1790697286279740 us; error 48 us; skew 500 ppm
I20260929 15:54:46.280148 15294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:54:46 dist-test-slave-r0ck krb5kdc[14216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:54:46.283252 15294 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:54:46.284703 15294 webserver.cc:460] Webserver started at http://127.0.97.62:45133/ using document root <none> and password file <none>
I20260929 15:54:46.285386 15294 fs_manager.cc:359] Metadata directory not provided
I20260929 15:54:46.285506 15294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:54:46.290570 15294 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:54:46.293138 15309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:46.294453 15294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:54:46.294616 15294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "b71b1df01d6748c1b6af1b4ec396e424"
format_stamp: "Formatted at 2026-09-29 15:54:40 on dist-test-slave-r0ck"
server_key: "5ad7883c77124b91a7f1c902381aabf1"
server_key_iv: "929a31de88676a3de6da37aaff8b32a6"
server_key_version: "encryptionkey@0"
I20260929 15:54:46.295559 15294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:54:46.317836 15294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:54:46.322115 15294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:54:46.322407 15294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:54:46.340960 15294 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45647
I20260929 15:54:46.341003 15360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45647 every 8 connection(s)
I20260929 15:54:46.342530 15294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:54:46.343350 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 15294
I20260929 15:54:46.344276 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:36887
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:45647
--webserver_port=34541
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:54:46.349460 15361 sys_catalog.cc:263] Verifying existing consensus state
I20260929 15:54:46.352523 15361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424: Bootstrap starting.
I20260929 15:54:46.372395 15361 log.cc:824] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424: Log is configured to *not* fsync() on all Append() calls
I20260929 15:54:46.387812 15361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 15:54:46.388322 15361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424: Bootstrap complete.
I20260929 15:54:46.397325 15361 raft_consensus.cc:348] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71b1df01d6748c1b6af1b4ec396e424" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45647 } }
I20260929 15:54:46.398270 15361 raft_consensus.cc:729] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b71b1df01d6748c1b6af1b4ec396e424, State: Initialized, Role: FOLLOWER
I20260929 15:54:46.398945 15361 consensus_queue.cc:260] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71b1df01d6748c1b6af1b4ec396e424" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45647 } }
I20260929 15:54:46.399230 15361 raft_consensus.cc:388] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:54:46.399360 15361 raft_consensus.cc:482] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:54:46.399550 15361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [term 2 FOLLOWER]: Advancing to term 3
I20260929 15:54:46.403388 15361 raft_consensus.cc:504] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71b1df01d6748c1b6af1b4ec396e424" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45647 } }
I20260929 15:54:46.403872 15361 leader_election.cc:302] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b71b1df01d6748c1b6af1b4ec396e424; no voters:
I20260929 15:54:46.404525 15361 leader_election.cc:288] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 15:54:46.404723 15365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 15:54:46.405875 15365 raft_consensus.cc:686] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [term 3 LEADER]: Becoming Leader. State: Replica: b71b1df01d6748c1b6af1b4ec396e424, State: Running, Role: LEADER
I20260929 15:54:46.406565 15361 sys_catalog.cc:564] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:54:46.406517 15365 consensus_queue.cc:237] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71b1df01d6748c1b6af1b4ec396e424" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45647 } }
I20260929 15:54:46.410322 15367 sys_catalog.cc:455] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b71b1df01d6748c1b6af1b4ec396e424. Latest consensus state: current_term: 3 leader_uuid: "b71b1df01d6748c1b6af1b4ec396e424" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71b1df01d6748c1b6af1b4ec396e424" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45647 } } }
I20260929 15:54:46.410742 15367 sys_catalog.cc:458] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [sys.catalog]: This master's current role is: LEADER
I20260929 15:54:46.411384 15366 sys_catalog.cc:455] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b71b1df01d6748c1b6af1b4ec396e424" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71b1df01d6748c1b6af1b4ec396e424" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45647 } } }
I20260929 15:54:46.411707 15366 sys_catalog.cc:458] T 00000000000000000000000000000000 P b71b1df01d6748c1b6af1b4ec396e424 [sys.catalog]: This master's current role is: LEADER
I20260929 15:54:46.423048 15375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:54:46.431332 15375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:54:46.433252 15375 catalog_manager.cc:1259] Loaded cluster ID: b3752e210cf14c11b1ef8533b38bb782
I20260929 15:54:46.433454 15375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 15:54:46.436060 15375 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
I20260929 15:54:46.438011 15382 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7fd105f769ad google::LogMessage::Fail() at ??:0
@ 0x7fd105f7988a google::LogMessage::SendToLog() at ??:0
@ 0x7fd105f764de google::LogMessage::Flush() at ??:0
@ 0x7fd105f77a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd11b9a9512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fd11b9a6976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd11ba2e610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fd11ba2e439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fd108bc2345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd108bc591c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fd108bc5799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fd108ba0eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd10be7a6db start_thread at ??:0
@ 0x7fd104e1061f clone at ??:0
W20260929 15:54:46.726574 15363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:54:46.726856 15363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:54:46.726951 15363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:54:46.734639 15363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:54:46.734759 15363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:54:46.734875 15363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:54:46.744733 15363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:36887
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=34541
--tserver_master_addrs=127.0.97.62:45647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:54:46.746068 15363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:54:46.747339 15363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:54:46.760679 15405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:46.760635 15403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:46.760912 15402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:54:46.766571 15363 server_base.cc:1034] running on GCE node
I20260929 15:54:46.767251 15363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:54:46.768146 15363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:54:46.769407 15363 hybrid_clock.cc:648] HybridClock initialized: now 1790697286769320 us; error 61 us; skew 500 ppm
I20260929 15:54:46.769714 15363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:54:46 dist-test-slave-r0ck krb5kdc[14216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:54:46.772680 15363 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:54:46.774111 15363 webserver.cc:460] Webserver started at http://127.0.97.1:34541/ using document root <none> and password file <none>
I20260929 15:54:46.774785 15363 fs_manager.cc:359] Metadata directory not provided
I20260929 15:54:46.774930 15363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:54:46.779969 15363 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 15:54:46.782706 15412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:46.783905 15363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:54:46.784082 15363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/wal
uuid: "40e6f6abaa6f4c6e8b23e1485098eb5a"
format_stamp: "Formatted at 2026-09-29 15:54:41 on dist-test-slave-r0ck"
server_key: "100e5ae7ebadf0a431ae8c4f322770d2"
server_key_iv: "c525c6b7c498ffd2ee115416e9ec929d"
server_key_version: "encryptionkey@0"
I20260929 15:54:46.784988 15363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:54:46.801304 15363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:54:46.804831 15363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:54:46.805095 15363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:54:46.806079 15363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:54:46.807744 15419 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260929 15:54:46.812871 15363 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 15:54:46.813004 15363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260929 15:54:46.813130 15363 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 15:54:46.816089 15363 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 15:54:46.816190 15363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260929 15:54:46.816480 15419 tablet_bootstrap.cc:492] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a: Bootstrap starting.
I20260929 15:54:46.824115 15419 log.cc:824] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a: Log is configured to *not* fsync() on all Append() calls
I20260929 15:54:46.838392 15419 tablet_bootstrap.cc:492] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 15:54:46.839401 15419 tablet_bootstrap.cc:492] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a: Bootstrap complete.
I20260929 15:54:46.839977 15419 ts_tablet_manager.cc:1397] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a: Time spent bootstrapping tablet: real 0.024s user 0.022s sys 0.000s
I20260929 15:54:46.850404 15419 raft_consensus.cc:348] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58ad1c480124e0c95775e4816ec6af3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42721 } } peers { permanent_uuid: "40e6f6abaa6f4c6e8b23e1485098eb5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36887 } } peers { permanent_uuid: "4e3912abb36648b9beed2653cd515807" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43165 } }
I20260929 15:54:46.851914 15419 raft_consensus.cc:729] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40e6f6abaa6f4c6e8b23e1485098eb5a, State: Initialized, Role: FOLLOWER
I20260929 15:54:46.852558 15419 consensus_queue.cc:260] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58ad1c480124e0c95775e4816ec6af3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42721 } } peers { permanent_uuid: "40e6f6abaa6f4c6e8b23e1485098eb5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36887 } } peers { permanent_uuid: "4e3912abb36648b9beed2653cd515807" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43165 } }
I20260929 15:54:46.854449 15419 ts_tablet_manager.cc:1428] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260929 15:54:46.856537 15363 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36887
I20260929 15:54:46.856552 15527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36887 every 8 connection(s)
I20260929 15:54:46.857769 15363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:54:46.862110 15530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:46.860309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.1:44737)
0929 15:54:46.861044 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:46.861227 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":321,"threads_started":1}
W20260929 15:54:46.862838 15528 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45647 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:45647: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
I20260929 15:54:46.863559 15530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:46.863275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.1:59837)
0929 15:54:46.863399 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:46.863464 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 15:54:46.864355 15530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:46.864058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.1:47551)
0929 15:54:46.864187 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:46.864251 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 15:54:46.864609 15528 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 15:54:46.867040 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 15363
I20260929 15:54:46.867661 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:42721
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:45647
--webserver_port=41911
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:54:47.183638 15531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:54:47.183893 15531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:54:47.184049 15531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:54:47.191305 15531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:54:47.191403 15531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:54:47.191602 15531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:54:47.200995 15531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:42721
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=41911
--tserver_master_addrs=127.0.97.62:45647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:54:47.202198 15531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:54:47.203401 15531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:54:47.215673 15536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:47.215709 15537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:47.215759 15539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:54:47.216609 15531 server_base.cc:1034] running on GCE node
I20260929 15:54:47.217099 15531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:54:47.218037 15531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:54:47.219246 15531 hybrid_clock.cc:648] HybridClock initialized: now 1790697287219193 us; error 24 us; skew 500 ppm
I20260929 15:54:47.219494 15531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:54:47 dist-test-slave-r0ck krb5kdc[14216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:54:47.222461 15531 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:54:47.223788 15531 webserver.cc:460] Webserver started at http://127.0.97.2:41911/ using document root <none> and password file <none>
I20260929 15:54:47.224376 15531 fs_manager.cc:359] Metadata directory not provided
I20260929 15:54:47.224470 15531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:54:47.229601 15531 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260929 15:54:47.232024 15546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:47.233281 15531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 15:54:47.233475 15531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/wal
uuid: "c58ad1c480124e0c95775e4816ec6af3"
format_stamp: "Formatted at 2026-09-29 15:54:41 on dist-test-slave-r0ck"
server_key: "73e61c8381046e8690ea9675601284d6"
server_key_iv: "8ec8c07f40200e240ddd43d5b32eb93f"
server_key_version: "encryptionkey@0"
I20260929 15:54:47.234380 15531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:54:47.284716 15531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:54:47.289140 15531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:54:47.289459 15531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:54:47.290391 15531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:54:47.291603 15531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:54:47.291694 15531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:47.291770 15531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:54:47.291823 15531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:47.324024 15531 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42721
I20260929 15:54:47.324150 15658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42721 every 8 connection(s)
I20260929 15:54:47.325388 15531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:54:47.329106 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 15531
I20260929 15:54:47.329865 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:43165
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:45647
--webserver_port=35317
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:54:47.330387 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:47.328220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.2:42339)
0929 15:54:47.328990 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:47.329289 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":260,"threads_started":1}
W20260929 15:54:47.330992 15659 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45647 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:45647: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
I20260929 15:54:47.331868 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:47.331463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.2:54575)
0929 15:54:47.331636 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:47.331723 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 15:54:47.332795 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:47.332393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.2:43213)
0929 15:54:47.332556 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:47.332648 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 15:54:47.333031 15659 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 15:54:47.643905 15662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:54:47.644200 15662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:54:47.644371 15662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:54:47.651438 15662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:54:47.651590 15662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:54:47.651760 15662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:54:47.660989 15662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:43165
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=35317
--tserver_master_addrs=127.0.97.62:45647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:54:47.662220 15662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:54:47.663422 15662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:54:47.675786 15667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:47.675872 15670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:54:47.675817 15668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:54:47.676599 15662 server_base.cc:1034] running on GCE node
I20260929 15:54:47.677095 15662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:54:47.678056 15662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:54:47.679270 15662 hybrid_clock.cc:648] HybridClock initialized: now 1790697287679214 us; error 51 us; skew 500 ppm
I20260929 15:54:47.679663 15662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:54:47 dist-test-slave-r0ck krb5kdc[14216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:54:47.682750 15662 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:54:47.684023 15662 webserver.cc:460] Webserver started at http://127.0.97.3:35317/ using document root <none> and password file <none>
I20260929 15:54:47.684625 15662 fs_manager.cc:359] Metadata directory not provided
I20260929 15:54:47.684718 15662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:54:47.689862 15662 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:54:47.692224 15677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:47.693495 15662 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 15:54:47.693681 15662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/wal
uuid: "4e3912abb36648b9beed2653cd515807"
format_stamp: "Formatted at 2026-09-29 15:54:42 on dist-test-slave-r0ck"
server_key: "672328c21ee8ba9890770a5c289c403c"
server_key_iv: "a5bd2d44cfa91404e0d3a320744b182c"
server_key_version: "encryptionkey@0"
I20260929 15:54:47.694589 15662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:54:47.710811 15662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:54:47.714956 15662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:54:47.715234 15662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:54:47.716159 15662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:54:47.717461 15662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:54:47.717594 15662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:47.717720 15662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:54:47.717811 15662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:54:47.750216 15662 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43165
I20260929 15:54:47.750279 15789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43165 every 8 connection(s)
I20260929 15:54:47.751664 15662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:54:47.756093 15792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:47.754393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.3:49785)
0929 15:54:47.755096 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:47.755272 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":271,"threads_started":1}
W20260929 15:54:47.756752 15790 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45647 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:45647: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
I20260929 15:54:47.757611 15792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:47.757225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.3:39157)
0929 15:54:47.757396 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:47.757468 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 15:54:47.758328 15792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:47.758033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.3:45047)
0929 15:54:47.758158 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:47.758227 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 15:54:47.758567 15790 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 15:54:47.761053 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 15662
I20260929 15:54:47.865971 15794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:47.865141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.1:60323)
0929 15:54:47.865686 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:47.865771 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":306,"threads_started":1}
I20260929 15:54:48.210391 15795 raft_consensus.cc:482] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:54:48.210693 15795 raft_consensus.cc:504] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58ad1c480124e0c95775e4816ec6af3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42721 } } peers { permanent_uuid: "40e6f6abaa6f4c6e8b23e1485098eb5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36887 } } peers { permanent_uuid: "4e3912abb36648b9beed2653cd515807" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43165 } }
I20260929 15:54:48.212299 15795 leader_election.cc:288] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c58ad1c480124e0c95775e4816ec6af3 (127.0.97.2:42721), 4e3912abb36648b9beed2653cd515807 (127.0.97.3:43165)
Sep 29 15:54:48 dist-test-slave-r0ck krb5kdc[14216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
Sep 29 15:54:48 dist-test-slave-r0ck krb5kdc[14216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 15:54:48.234661 15798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:48.212813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54237 (local address 127.0.97.2:42721)
0929 15:54:48.215231 (+ 2418us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:48.215244 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:48.215278 (+ 34us) server_negotiation.cc:407] Connection header received
0929 15:54:48.217300 (+ 2022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:48.217308 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:48.217709 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:48.218067 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:48.219652 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:48.220798 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:48.222596 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:48.223206 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:48.223392 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:48.227771 (+ 4379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:48.227778 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:48.227790 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:48.227824 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:48.230956 (+ 3132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:48.231728 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:48.231734 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:48.231739 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:48.231882 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:48.232324 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:48.232328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:48.232330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:48.232825 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:48.233034 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:48.233870 (+ 836us) server_negotiation.cc:299] Negotiation successful
0929 15:54:48.234219 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2260,"thread_start_us":2131,"threads_started":1}
I20260929 15:54:48.234659 15794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:48.212630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42721 (local address 127.0.97.1:54237)
0929 15:54:48.212966 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:48.213004 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 15:54:48.213705 (+ 701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:54:48.218175 (+ 4470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:54:48.218183 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:54:48.218797 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:54:48.219453 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:48.219472 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:48.221005 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:48.221009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:48.222449 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:48.222461 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:48.223415 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:48.223422 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:48.223730 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:48.224490 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:54:48.224524 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:54:48.227506 (+ 2982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:54:48.231094 (+ 3588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:48.231105 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:48.231109 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:48.231587 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:48.231997 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:48.232004 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:48.232008 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:48.232201 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:48.232969 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:54:48.232977 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:54:48.233329 (+ 352us) client_negotiation.cc:769] Sending connection context
0929 15:54:48.233656 (+ 327us) client_negotiation.cc:240] Negotiation successful
0929 15:54:48.234219 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":281,"spinlock_wait_cycles":16640}
I20260929 15:54:48.234676 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:48.212873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41591 (local address 127.0.97.3:43165)
0929 15:54:48.214949 (+ 2076us) server_negotiation.cc:206] Beginning negotiation
0929 15:54:48.214966 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 15:54:48.215018 (+ 52us) server_negotiation.cc:407] Connection header received
0929 15:54:48.217180 (+ 2162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:54:48.217193 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:54:48.217710 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:54:48.218065 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:54:48.219655 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:48.220804 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:48.222593 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:54:48.223213 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:54:48.223436 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:48.227636 (+ 4200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:54:48.227645 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:54:48.227658 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:54:48.227692 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:54:48.230862 (+ 3170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:48.231721 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:48.231729 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:48.231734 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:48.231885 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:54:48.232336 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:54:48.232341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:54:48.232346 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:54:48.232804 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:54:48.233001 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 15:54:48.233918 (+ 917us) server_negotiation.cc:299] Negotiation successful
0929 15:54:48.234249 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1894,"thread_start_us":1788,"threads_started":1}
I20260929 15:54:48.234809 15796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:48.212772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43165 (local address 127.0.97.1:41591)
0929 15:54:48.214625 (+ 1853us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:48.214645 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:54:48.214780 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:54:48.218616 (+ 3836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:54:48.218619 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:54:48.219117 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:54:48.219456 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:48.219475 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:48.220993 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:48.220997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:48.222451 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:54:48.222463 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:54:48.223439 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:54:48.223445 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:54:48.223729 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:54:48.224531 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:54:48.224549 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:54:48.227270 (+ 2721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:54:48.231023 (+ 3753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:48.231032 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:48.231045 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:48.231572 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:48.232012 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:54:48.232017 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:54:48.232021 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:54:48.232222 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:54:48.232971 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:54:48.232978 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:54:48.233328 (+ 350us) client_negotiation.cc:769] Sending connection context
0929 15:54:48.233656 (+ 328us) client_negotiation.cc:240] Negotiation successful
0929 15:54:48.234397 (+ 741us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1686,"mutex_wait_us":438,"spinlock_wait_cycles":15872,"thread_start_us":1447,"threads_started":1}
W20260929 15:54:48.238022 15414 leader_election.cc:341] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4e3912abb36648b9beed2653cd515807 (127.0.97.3:43165): Not found: Tablet not found: 049feb5ebe714e37b137ae053dc53abd
W20260929 15:54:48.238265 15415 leader_election.cc:341] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c58ad1c480124e0c95775e4816ec6af3 (127.0.97.2:42721): Not found: Tablet not found: 049feb5ebe714e37b137ae053dc53abd
I20260929 15:54:48.238432 15415 leader_election.cc:302] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40e6f6abaa6f4c6e8b23e1485098eb5a; no voters: 4e3912abb36648b9beed2653cd515807, c58ad1c480124e0c95775e4816ec6af3
I20260929 15:54:48.238852 15795 raft_consensus.cc:2726] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 15:54:48.334479 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:48.333593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.2:46655)
0929 15:54:48.334071 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:48.334200 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":231,"threads_started":1}
I20260929 15:54:48.759774 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:48.759052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.3:58221)
0929 15:54:48.759464 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:48.759576 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":212,"threads_started":1}
I20260929 15:54:48.867616 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:48.866894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.1:43123)
0929 15:54:48.867344 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:48.867423 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":221,"threads_started":1}
I20260929 15:54:49.335960 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:49.335372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.2:42873)
0929 15:54:49.335748 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:49.335818 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":199,"threads_started":1}
I20260929 15:54:49.761320 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:49.760635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.3:49647)
0929 15:54:49.761065 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:49.761147 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":230,"threads_started":1}
I20260929 15:54:49.869390 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:49.868575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.1:39413)
0929 15:54:49.869082 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:49.869174 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":275,"threads_started":1}
I20260929 15:54:49.998556 15805 raft_consensus.cc:482] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:54:49.998755 15805 raft_consensus.cc:504] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58ad1c480124e0c95775e4816ec6af3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42721 } } peers { permanent_uuid: "40e6f6abaa6f4c6e8b23e1485098eb5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36887 } } peers { permanent_uuid: "4e3912abb36648b9beed2653cd515807" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43165 } }
I20260929 15:54:49.999439 15805 leader_election.cc:288] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c58ad1c480124e0c95775e4816ec6af3 (127.0.97.2:42721), 4e3912abb36648b9beed2653cd515807 (127.0.97.3:43165)
W20260929 15:54:50.000561 15414 leader_election.cc:341] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4e3912abb36648b9beed2653cd515807 (127.0.97.3:43165): Not found: Tablet not found: 049feb5ebe714e37b137ae053dc53abd
W20260929 15:54:50.000756 15415 leader_election.cc:341] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c58ad1c480124e0c95775e4816ec6af3 (127.0.97.2:42721): Not found: Tablet not found: 049feb5ebe714e37b137ae053dc53abd
I20260929 15:54:50.000852 15415 leader_election.cc:302] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40e6f6abaa6f4c6e8b23e1485098eb5a; no voters: 4e3912abb36648b9beed2653cd515807, c58ad1c480124e0c95775e4816ec6af3
I20260929 15:54:50.001082 15805 raft_consensus.cc:2726] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 15:54:50.337728 15806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:50.336794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.2:40853)
0929 15:54:50.337411 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:50.337552 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":242,"threads_started":1}
I20260929 15:54:50.762852 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:50.762195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.3:50999)
0929 15:54:50.762611 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:50.762684 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":229,"threads_started":1}
I20260929 15:54:50.871066 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:50.870297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.1:38569)
0929 15:54:50.870812 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:50.870889 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":314,"threads_started":1}
I20260929 15:54:51.339355 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:51.338578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.2:33957)
0929 15:54:51.339076 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:51.339181 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":232,"threads_started":1}
I20260929 15:54:51.727409 15810 raft_consensus.cc:482] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:54:51.727634 15810 raft_consensus.cc:504] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58ad1c480124e0c95775e4816ec6af3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42721 } } peers { permanent_uuid: "40e6f6abaa6f4c6e8b23e1485098eb5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36887 } } peers { permanent_uuid: "4e3912abb36648b9beed2653cd515807" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43165 } }
I20260929 15:54:51.728379 15810 leader_election.cc:288] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c58ad1c480124e0c95775e4816ec6af3 (127.0.97.2:42721), 4e3912abb36648b9beed2653cd515807 (127.0.97.3:43165)
W20260929 15:54:51.729323 15415 leader_election.cc:341] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c58ad1c480124e0c95775e4816ec6af3 (127.0.97.2:42721): Not found: Tablet not found: 049feb5ebe714e37b137ae053dc53abd
W20260929 15:54:51.729578 15414 leader_election.cc:341] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4e3912abb36648b9beed2653cd515807 (127.0.97.3:43165): Not found: Tablet not found: 049feb5ebe714e37b137ae053dc53abd
I20260929 15:54:51.729775 15414 leader_election.cc:302] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40e6f6abaa6f4c6e8b23e1485098eb5a; no voters: 4e3912abb36648b9beed2653cd515807, c58ad1c480124e0c95775e4816ec6af3
I20260929 15:54:51.730039 15810 raft_consensus.cc:2726] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 15:54:51.764600 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:51.763790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.3:50945)
0929 15:54:51.764288 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:51.764364 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":268,"threads_started":1}
I20260929 15:54:51.873031 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:51.872132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.1:44917)
0929 15:54:51.872671 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:51.872766 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":237,"threads_started":1}
I20260929 15:54:52.340826 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:52.340196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.2:54813)
0929 15:54:52.340601 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:52.340676 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":219,"threads_started":1}
I20260929 15:54:52.766518 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:52.765681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.3:44625)
0929 15:54:52.766236 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:52.766327 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":310,"threads_started":1}
I20260929 15:54:52.874642 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:52.873949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.1:35769)
0929 15:54:52.874386 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:52.874482 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":224,"threads_started":1}
I20260929 15:54:53.342561 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:53.341777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.2:48003)
0929 15:54:53.342296 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:53.342376 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":296,"threads_started":1}
I20260929 15:54:53.768051 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:53.767338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.3:58657)
0929 15:54:53.767799 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:53.767877 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":249,"threads_started":1}
I20260929 15:54:53.876197 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:53.875493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.1:36795)
0929 15:54:53.875911 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:53.875987 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":232,"threads_started":1}
I20260929 15:54:54.344312 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:54.343517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.2:53091)
0929 15:54:54.344029 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:54.344127 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":229,"threads_started":1}
I20260929 15:54:54.769868 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:54.768992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.3:57015)
0929 15:54:54.769574 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:54.769657 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":244,"threads_started":1}
I20260929 15:54:54.878013 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:54.877252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.1:39903)
0929 15:54:54.877755 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:54.877839 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":281,"threads_started":1}
I20260929 15:54:55.345892 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:55.345163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.2:34831)
0929 15:54:55.345635 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:55.345712 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":267,"threads_started":1}
I20260929 15:54:55.771514 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:55.770782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.3:54929)
0929 15:54:55.771274 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:55.771350 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":257,"threads_started":1}
I20260929 15:54:55.879667 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:55.878923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.1:47179)
0929 15:54:55.879378 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:55.879490 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":224,"threads_started":1}
I20260929 15:54:56.045434 15825 raft_consensus.cc:482] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:54:56.045653 15825 raft_consensus.cc:504] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58ad1c480124e0c95775e4816ec6af3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42721 } } peers { permanent_uuid: "40e6f6abaa6f4c6e8b23e1485098eb5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36887 } } peers { permanent_uuid: "4e3912abb36648b9beed2653cd515807" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43165 } }
I20260929 15:54:56.046422 15825 leader_election.cc:288] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c58ad1c480124e0c95775e4816ec6af3 (127.0.97.2:42721), 4e3912abb36648b9beed2653cd515807 (127.0.97.3:43165)
W20260929 15:54:56.047358 15415 leader_election.cc:341] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c58ad1c480124e0c95775e4816ec6af3 (127.0.97.2:42721): Not found: Tablet not found: 049feb5ebe714e37b137ae053dc53abd
W20260929 15:54:56.047554 15414 leader_election.cc:341] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4e3912abb36648b9beed2653cd515807 (127.0.97.3:43165): Not found: Tablet not found: 049feb5ebe714e37b137ae053dc53abd
I20260929 15:54:56.047653 15414 leader_election.cc:302] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40e6f6abaa6f4c6e8b23e1485098eb5a; no voters: 4e3912abb36648b9beed2653cd515807, c58ad1c480124e0c95775e4816ec6af3
I20260929 15:54:56.047873 15825 raft_consensus.cc:2726] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 15:54:56.347422 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:56.346726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.2:46445)
0929 15:54:56.347198 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:56.347271 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":227,"threads_started":1}
I20260929 15:54:56.773025 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:56.772329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.3:57433)
0929 15:54:56.772782 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:56.772866 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":244,"threads_started":1}
I20260929 15:54:56.881402 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:56.880647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.1:54249)
0929 15:54:56.881081 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:56.881170 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":222,"threads_started":1}
I20260929 15:54:57.348948 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:57.348266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.2:35621)
0929 15:54:57.348700 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:57.348777 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":224,"threads_started":1}
I20260929 15:54:57.774626 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:57.773930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.3:53315)
0929 15:54:57.774357 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:57.774432 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":218,"threads_started":1}
I20260929 15:54:57.883260 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:57.882416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.1:39543)
0929 15:54:57.882906 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:57.883037 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":245,"threads_started":1}
I20260929 15:54:58.320859 15832 raft_consensus.cc:482] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:54:58.321080 15832 raft_consensus.cc:504] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58ad1c480124e0c95775e4816ec6af3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42721 } } peers { permanent_uuid: "40e6f6abaa6f4c6e8b23e1485098eb5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36887 } } peers { permanent_uuid: "4e3912abb36648b9beed2653cd515807" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43165 } }
I20260929 15:54:58.321797 15832 leader_election.cc:288] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c58ad1c480124e0c95775e4816ec6af3 (127.0.97.2:42721), 4e3912abb36648b9beed2653cd515807 (127.0.97.3:43165)
W20260929 15:54:58.322901 15414 leader_election.cc:341] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4e3912abb36648b9beed2653cd515807 (127.0.97.3:43165): Not found: Tablet not found: 049feb5ebe714e37b137ae053dc53abd
W20260929 15:54:58.323117 15415 leader_election.cc:341] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c58ad1c480124e0c95775e4816ec6af3 (127.0.97.2:42721): Not found: Tablet not found: 049feb5ebe714e37b137ae053dc53abd
I20260929 15:54:58.323210 15415 leader_election.cc:302] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40e6f6abaa6f4c6e8b23e1485098eb5a; no voters: 4e3912abb36648b9beed2653cd515807, c58ad1c480124e0c95775e4816ec6af3
I20260929 15:54:58.323421 15832 raft_consensus.cc:2726] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 15:54:58.350538 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:58.349858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.2:50831)
0929 15:54:58.350295 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:58.350384 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":223,"threads_started":1}
I20260929 15:54:58.776273 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:58.775558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.3:45491)
0929 15:54:58.776047 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:58.776127 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":246,"threads_started":1}
I20260929 15:54:58.884927 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:58.884204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.1:40557)
0929 15:54:58.884658 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:58.884743 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":258,"threads_started":1}
I20260929 15:54:59.352162 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:59.351445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.2:43199)
0929 15:54:59.351896 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:59.351974 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":250,"threads_started":1}
I20260929 15:54:59.778141 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:59.777392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.3:40021)
0929 15:54:59.777874 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:59.777959 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":238,"threads_started":1}
I20260929 15:54:59.886719 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:54:59.885884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.1:33681)
0929 15:54:59.886422 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0929 15:54:59.886538 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":329,"threads_started":1}
I20260929 15:55:00.199124 15839 raft_consensus.cc:482] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:55:00.199388 15839 raft_consensus.cc:504] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58ad1c480124e0c95775e4816ec6af3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42721 } } peers { permanent_uuid: "40e6f6abaa6f4c6e8b23e1485098eb5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36887 } } peers { permanent_uuid: "4e3912abb36648b9beed2653cd515807" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43165 } }
I20260929 15:55:00.200165 15839 leader_election.cc:288] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c58ad1c480124e0c95775e4816ec6af3 (127.0.97.2:42721), 4e3912abb36648b9beed2653cd515807 (127.0.97.3:43165)
W20260929 15:55:00.201122 15415 leader_election.cc:341] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c58ad1c480124e0c95775e4816ec6af3 (127.0.97.2:42721): Not found: Tablet not found: 049feb5ebe714e37b137ae053dc53abd
W20260929 15:55:00.201300 15414 leader_election.cc:341] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4e3912abb36648b9beed2653cd515807 (127.0.97.3:43165): Not found: Tablet not found: 049feb5ebe714e37b137ae053dc53abd
I20260929 15:55:00.201464 15414 leader_election.cc:302] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40e6f6abaa6f4c6e8b23e1485098eb5a; no voters: 4e3912abb36648b9beed2653cd515807, c58ad1c480124e0c95775e4816ec6af3
I20260929 15:55:00.201823 15839 raft_consensus.cc:2726] T 049feb5ebe714e37b137ae053dc53abd P 40e6f6abaa6f4c6e8b23e1485098eb5a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 15:55:00.353917 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:00.353100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.2:58463)
0929 15:55:00.353668 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:00.353748 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":337,"threads_started":1}
I20260929 15:55:00.779881 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:00.779100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.3:54415)
0929 15:55:00.779575 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:00.779692 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":231,"threads_started":1}
I20260929 15:55:00.888607 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:00.887714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.1:37395)
0929 15:55:00.888248 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:00.888368 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":311,"threads_started":1}
I20260929 15:55:01.355809 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:01.355020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.2:36151)
0929 15:55:01.355518 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:01.355615 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":262,"threads_started":1}
I20260929 15:55:01.781729 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:01.780916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.3:50357)
0929 15:55:01.781448 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:01.781529 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":246,"threads_started":1}
I20260929 15:55:01.890552 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:01.889802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.1:51087)
0929 15:55:01.890276 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:01.890360 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":236,"threads_started":1}
I20260929 15:55:02.358168 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:02.357462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.2:54831)
0929 15:55:02.357893 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:02.357983 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":223,"threads_started":1}
I20260929 15:55:02.765992 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 15363
I20260929 15:55:02.781855 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:02.767715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52588 (local address 127.0.97.1:36887)
0929 15:55:02.768787 (+ 1072us) server_negotiation.cc:206] Beginning negotiation
0929 15:55:02.768797 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:55:02.768832 (+ 35us) server_negotiation.cc:407] Connection header received
0929 15:55:02.768927 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:55:02.768940 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:55:02.769264 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:55:02.769450 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:55:02.771063 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:02.772002 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:02.773236 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:02.773874 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:02.774028 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:02.776495 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:55:02.776516 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:55:02.776531 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:55:02.776561 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:55:02.779144 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:55:02.779808 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:55:02.779817 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:55:02.779820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:55:02.779953 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:55:02.780313 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:55:02.780319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:55:02.780323 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:55:02.780656 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:55:02.780933 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0929 15:55:02.781237 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 15:55:02.781524 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":869,"thread_start_us":748,"threads_started":1}
I20260929 15:55:02.783188 15523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:55:02.783411 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:02.782670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45647 (local address 127.0.97.3:40243)
0929 15:55:02.783113 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:02.783187 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":234,"threads_started":1}
I20260929 15:55:02.872519 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 15363
I20260929 15:55:02.890071 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 15531
I20260929 15:55:02.903029 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:02.890815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41646 (local address 127.0.97.2:42721)
0929 15:55:02.891290 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0929 15:55:02.891294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:55:02.891310 (+ 16us) server_negotiation.cc:407] Connection header received
0929 15:55:02.891365 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:55:02.891368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:55:02.891424 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:55:02.891514 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:55:02.892765 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:02.893589 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:02.894861 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:02.895257 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:02.895345 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:02.898005 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:55:02.898010 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:55:02.898014 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:55:02.898043 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:55:02.900684 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:55:02.901294 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:55:02.901297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:55:02.901299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:55:02.901466 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:55:02.901775 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:55:02.901778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:55:02.901780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:55:02.902033 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:55:02.902228 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 15:55:02.902498 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 15:55:02.902670 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":266,"threads_started":1}
I20260929 15:55:02.903730 15654 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:55:02.985868 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 15531
I20260929 15:55:03.004367 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 15662
I20260929 15:55:03.018755 15852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:03.005266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46130 (local address 127.0.97.3:43165)
0929 15:55:03.005916 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0929 15:55:03.005921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:55:03.005938 (+ 17us) server_negotiation.cc:407] Connection header received
0929 15:55:03.005989 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:55:03.005993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:55:03.006058 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:55:03.006164 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:55:03.007727 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:03.008530 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:03.009938 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:03.010412 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:03.010521 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:03.013321 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:55:03.013326 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:55:03.013329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:55:03.013442 (+ 113us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:55:03.016173 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:55:03.016886 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:55:03.016890 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:55:03.016892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:55:03.017025 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:55:03.017433 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:55:03.017436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:55:03.017438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:55:03.017714 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:55:03.017928 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 15:55:03.018202 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 15:55:03.018366 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":392,"threads_started":1}
I20260929 15:55:03.019498 15785 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:55:03.101111 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 15662
2026-09-29T15:55:03Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24234 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:55:03 dist-test-slave-r0ck krb5kdc[15856](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:55:03 dist-test-slave-r0ck krb5kdc[15856](info): set up 2 sockets
Sep 29 15:55:03 dist-test-slave-r0ck krb5kdc[15856](info): commencing operation
krb5kdc: starting...
W20260929 15:55:04.206043 388 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:55:04 dist-test-slave-r0ck krb5kdc[15856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:55:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:55:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:55:04.437731 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38491
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:55:04.755944 15869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:55:04.756220 15869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:55:04.756307 15869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:55:04.763357 15869 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 15:55:04.763459 15869 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 15:55:04.763497 15869 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 15:55:04.763604 15869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:55:04.763682 15869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:55:04.763801 15869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:55:04.763852 15869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:55:04.763931 15869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:55:04.773519 15869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:55:04.774777 15869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:55:04.776041 15869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:55:04.787484 15877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:55:04.787533 15874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:55:04.787533 15875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:55:04.788415 15869 server_base.cc:1034] running on GCE node
I20260929 15:55:04.789165 15869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:55:04.790517 15869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:55:04.791762 15869 hybrid_clock.cc:648] HybridClock initialized: now 1790697304791701 us; error 58 us; skew 500 ppm
I20260929 15:55:04.792092 15869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:55:04 dist-test-slave-r0ck krb5kdc[15856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:55:04.795157 15869 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:55:04.796478 15869 webserver.cc:460] Webserver started at http://127.0.97.62:34989/ using document root <none> and password file <none>
I20260929 15:55:04.797094 15869 fs_manager.cc:359] Metadata directory not provided
I20260929 15:55:04.797175 15869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:55:04.797487 15869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:55:04.800300 15869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "29da14e05d2b468ba3c91329fd95aff9"
format_stamp: "Formatted at 2026-09-29 15:55:04 on dist-test-slave-r0ck"
server_key: "f500f5b7e97e070a6e58e107e4e04045"
server_key_iv: "3bbd59075cf52066af916acb4c690389"
server_key_version: "encryptionkey@0"
I20260929 15:55:04.800995 15869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "29da14e05d2b468ba3c91329fd95aff9"
format_stamp: "Formatted at 2026-09-29 15:55:04 on dist-test-slave-r0ck"
server_key: "f500f5b7e97e070a6e58e107e4e04045"
server_key_iv: "3bbd59075cf52066af916acb4c690389"
server_key_version: "encryptionkey@0"
I20260929 15:55:04.805986 15869 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 15:55:04.809185 15884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:55:04.810539 15869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:55:04.810731 15869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "29da14e05d2b468ba3c91329fd95aff9"
format_stamp: "Formatted at 2026-09-29 15:55:04 on dist-test-slave-r0ck"
server_key: "f500f5b7e97e070a6e58e107e4e04045"
server_key_iv: "3bbd59075cf52066af916acb4c690389"
server_key_version: "encryptionkey@0"
I20260929 15:55:04.810949 15869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:55:04.829873 15869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:55:04.833879 15869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:55:04.834154 15869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:55:04.852744 15869 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38491
I20260929 15:55:04.852779 15935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38491 every 8 connection(s)
I20260929 15:55:04.854162 15869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:55:04.857990 15936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:55:04.860090 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 15869
I20260929 15:55:04.860303 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:55:04.860988 388 external_mini_cluster.cc:1428] Setting key df2adf9dc3542d204472cb2dceca6a6f
I20260929 15:55:04.871130 15936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9: Bootstrap starting.
I20260929 15:55:04.874461 15936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9: Neither blocks nor log segments found. Creating new log.
I20260929 15:55:04.875411 15936 log.cc:824] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9: Log is configured to *not* fsync() on all Append() calls
Sep 29 15:55:04 dist-test-slave-r0ck krb5kdc[15856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:55:04.878106 15936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9: No bootstrap required, opened a new log
I20260929 15:55:04.885993 15939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:04.863034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55502 (local address 127.0.97.62:38491)
0929 15:55:04.863933 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0929 15:55:04.863948 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 15:55:04.863998 (+ 50us) server_negotiation.cc:407] Connection header received
0929 15:55:04.866360 (+ 2362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:55:04.866387 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:55:04.866869 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:55:04.867307 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:55:04.868717 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:04.870639 (+ 1922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:04.872069 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:04.873145 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:04.873316 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:04.877469 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:55:04.877476 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:55:04.877489 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:55:04.877523 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:55:04.881465 (+ 3942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:55:04.882217 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:55:04.882226 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:55:04.882233 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:55:04.882434 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:55:04.882807 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:55:04.882813 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:55:04.882816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:55:04.883496 (+ 680us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:55:04.883782 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0929 15:55:04.884172 (+ 390us) server_negotiation.cc:299] Negotiation successful
0929 15:55:04.884613 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":394,"threads_started":1}
I20260929 15:55:04.888223 15936 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29da14e05d2b468ba3c91329fd95aff9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38491 } }
I20260929 15:55:04.888605 15936 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:55:04.888724 15936 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29da14e05d2b468ba3c91329fd95aff9, State: Initialized, Role: FOLLOWER
I20260929 15:55:04.889307 15936 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29da14e05d2b468ba3c91329fd95aff9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38491 } }
I20260929 15:55:04.889582 15936 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:55:04.889681 15936 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:55:04.889796 15936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:55:04.891844 15936 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29da14e05d2b468ba3c91329fd95aff9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38491 } }
I20260929 15:55:04.892266 15936 leader_election.cc:302] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29da14e05d2b468ba3c91329fd95aff9; no voters:
I20260929 15:55:04.892787 15936 leader_election.cc:288] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:55:04.892930 15941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:55:04.893649 15941 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [term 1 LEADER]: Becoming Leader. State: Replica: 29da14e05d2b468ba3c91329fd95aff9, State: Running, Role: LEADER
I20260929 15:55:04.894181 15941 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29da14e05d2b468ba3c91329fd95aff9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38491 } }
I20260929 15:55:04.895053 15936 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:55:04.896953 15942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29da14e05d2b468ba3c91329fd95aff9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29da14e05d2b468ba3c91329fd95aff9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38491 } } }
I20260929 15:55:04.897446 15942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [sys.catalog]: This master's current role is: LEADER
I20260929 15:55:04.897855 15943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29da14e05d2b468ba3c91329fd95aff9. Latest consensus state: current_term: 1 leader_uuid: "29da14e05d2b468ba3c91329fd95aff9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29da14e05d2b468ba3c91329fd95aff9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38491 } } }
I20260929 15:55:04.899220 15943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [sys.catalog]: This master's current role is: LEADER
I20260929 15:55:04.903338 15950 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:55:04.908721 15950 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:55:04.918303 15950 catalog_manager.cc:1347] Generated new cluster ID: b75c332d7dba4033b10c80d75da9bcb8
I20260929 15:55:04.918447 15950 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:55:04.939256 15950 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:55:04.940582 15950 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:55:04.953519 15950 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9: Generated new TSK 0
I20260929 15:55:04.954361 15950 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 15:55:05.014022 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38491
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:55:05.341777 15962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:55:05.342051 15962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:55:05.342145 15962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:55:05.349164 15962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:55:05.349258 15962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:55:05.349435 15962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:55:05.358851 15962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38491
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:55:05.360167 15962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:55:05.361423 15962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:55:05.374197 15970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:55:05.374118 15968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:55:05.374162 15967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:55:05.374881 15962 server_base.cc:1034] running on GCE node
I20260929 15:55:05.375492 15962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:55:05.376389 15962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:55:05.377630 15962 hybrid_clock.cc:648] HybridClock initialized: now 1790697305377558 us; error 55 us; skew 500 ppm
I20260929 15:55:05.377954 15962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:55:05 dist-test-slave-r0ck krb5kdc[15856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:55:05.386312 15962 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:55:05.387885 15962 webserver.cc:460] Webserver started at http://127.0.97.1:36367/ using document root <none> and password file <none>
I20260929 15:55:05.388564 15962 fs_manager.cc:359] Metadata directory not provided
I20260929 15:55:05.388707 15962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:55:05.389008 15962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:55:05.391857 15962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/data/instance:
uuid: "725fc31fc8b241cb8990f3e8cfb8ac44"
format_stamp: "Formatted at 2026-09-29 15:55:05 on dist-test-slave-r0ck"
server_key: "51807be11621fed588914153c42c4e30"
server_key_iv: "9bb144e888d933f0b4c26c8ce5790384"
server_key_version: "encryptionkey@0"
I20260929 15:55:05.392608 15962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "725fc31fc8b241cb8990f3e8cfb8ac44"
format_stamp: "Formatted at 2026-09-29 15:55:05 on dist-test-slave-r0ck"
server_key: "51807be11621fed588914153c42c4e30"
server_key_iv: "9bb144e888d933f0b4c26c8ce5790384"
server_key_version: "encryptionkey@0"
I20260929 15:55:05.397672 15962 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 15:55:05.401206 15977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:55:05.402655 15962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:55:05.402834 15962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/wal
uuid: "725fc31fc8b241cb8990f3e8cfb8ac44"
format_stamp: "Formatted at 2026-09-29 15:55:05 on dist-test-slave-r0ck"
server_key: "51807be11621fed588914153c42c4e30"
server_key_iv: "9bb144e888d933f0b4c26c8ce5790384"
server_key_version: "encryptionkey@0"
I20260929 15:55:05.403033 15962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:55:05.434947 15962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:55:05.438943 15962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:55:05.439240 15962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:55:05.440198 15962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:55:05.441641 15962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:55:05.441744 15962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:55:05.441897 15962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:55:05.441978 15962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:55:05.475237 15962 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35363
I20260929 15:55:05.475288 16089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35363 every 8 connection(s)
I20260929 15:55:05.476552 15962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:55:05.482102 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 15962
I20260929 15:55:05.482324 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:55:05.483091 388 external_mini_cluster.cc:1428] Setting key 7baa51cb3c0bd4ffa2bb6b79ee06641a
Sep 29 15:55:05 dist-test-slave-r0ck krb5kdc[15856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:55:05.498579 16092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:05.479105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60919 (local address 127.0.97.62:38491)
0929 15:55:05.479575 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0929 15:55:05.479579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:55:05.480316 (+ 737us) server_negotiation.cc:407] Connection header received
0929 15:55:05.482488 (+ 2172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:55:05.482494 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:55:05.482577 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:55:05.482669 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:55:05.484560 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:05.485627 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:05.488171 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:05.488611 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:05.488721 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:05.492830 (+ 4109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:55:05.492835 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:55:05.492838 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:55:05.492877 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:55:05.495522 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:55:05.496235 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:55:05.496243 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:55:05.496247 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:55:05.496375 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:55:05.496803 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:55:05.496808 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:55:05.496812 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:55:05.497080 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:55:05.497255 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 15:55:05.497989 (+ 734us) server_negotiation.cc:299] Negotiation successful
0929 15:55:05.498205 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":254,"threads_started":1}
I20260929 15:55:05.499655 16093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:05.479485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.1:60919)
0929 15:55:05.480150 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:05.480186 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 15:55:05.482122 (+ 1936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:55:05.482955 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:55:05.482969 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:55:05.483701 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:55:05.484385 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:55:05.484403 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:55:05.485790 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:55:05.485797 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:55:05.487975 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:55:05.488005 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:55:05.488847 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:55:05.488856 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:55:05.489174 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:05.489905 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:55:05.489937 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:55:05.492609 (+ 2672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:55:05.495692 (+ 3083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:55:05.495701 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:55:05.495716 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:55:05.496115 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:55:05.496488 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:55:05.496493 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:55:05.496497 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:55:05.496689 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:55:05.497212 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:55:05.497220 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:55:05.497613 (+ 393us) client_negotiation.cc:769] Sending connection context
0929 15:55:05.497869 (+ 256us) client_negotiation.cc:240] Negotiation successful
0929 15:55:05.498434 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":282,"threads_started":1}
I20260929 15:55:05.501210 16090 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38491
I20260929 15:55:05.501554 16090 heartbeater.cc:464] Registering TS with master...
I20260929 15:55:05.502408 16090 heartbeater.cc:511] Master 127.0.97.62:38491 requested a full tablet report, sending...
I20260929 15:55:05.504897 15901 ts_manager.cc:194] Registered new tserver with Master: 725fc31fc8b241cb8990f3e8cfb8ac44 (127.0.97.1:35363)
I20260929 15:55:05.507128 15901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:60919
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 15:55:05.537642 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38491
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:55:05.854080 16096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:55:05.854326 16096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:55:05.854494 16096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:55:05.861613 16096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:55:05.861711 16096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:55:05.861920 16096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:55:05.871409 16096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38491
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:55:05.872611 16096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:55:05.873869 16096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:55:05.886883 16101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:55:05.886826 16102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:55:05.886824 16104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:55:05.887750 16096 server_base.cc:1034] running on GCE node
I20260929 15:55:05.888330 16096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:55:05.889241 16096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:55:05.890476 16096 hybrid_clock.cc:648] HybridClock initialized: now 1790697305890439 us; error 64 us; skew 500 ppm
I20260929 15:55:05.890779 16096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:55:05 dist-test-slave-r0ck krb5kdc[15856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:55:05.893963 16096 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:55:05.895511 16096 webserver.cc:460] Webserver started at http://127.0.97.2:33541/ using document root <none> and password file <none>
I20260929 15:55:05.896296 16096 fs_manager.cc:359] Metadata directory not provided
I20260929 15:55:05.896441 16096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:55:05.896759 16096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:55:05.899748 16096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/data/instance:
uuid: "b2cce6a0fc254095826871e8d5a0a0fc"
format_stamp: "Formatted at 2026-09-29 15:55:05 on dist-test-slave-r0ck"
server_key: "724f69044ce0f997bb65f206403f49ba"
server_key_iv: "877f61eb7d1787f7c87f742607d0b2ac"
server_key_version: "encryptionkey@0"
I20260929 15:55:05.900516 16096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b2cce6a0fc254095826871e8d5a0a0fc"
format_stamp: "Formatted at 2026-09-29 15:55:05 on dist-test-slave-r0ck"
server_key: "724f69044ce0f997bb65f206403f49ba"
server_key_iv: "877f61eb7d1787f7c87f742607d0b2ac"
server_key_version: "encryptionkey@0"
I20260929 15:55:05.905519 16096 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 15:55:05.908892 16111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:55:05.910533 16096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:55:05.910709 16096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/wal
uuid: "b2cce6a0fc254095826871e8d5a0a0fc"
format_stamp: "Formatted at 2026-09-29 15:55:05 on dist-test-slave-r0ck"
server_key: "724f69044ce0f997bb65f206403f49ba"
server_key_iv: "877f61eb7d1787f7c87f742607d0b2ac"
server_key_version: "encryptionkey@0"
I20260929 15:55:05.910910 16096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:55:05.924009 16096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:55:05.928462 16096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:55:05.928753 16096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:55:05.929774 16096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:55:05.931064 16096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:55:05.931145 16096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:55:05.931285 16096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:55:05.931347 16096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:55:05.963390 16096 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37361
I20260929 15:55:05.963482 16223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37361 every 8 connection(s)
I20260929 15:55:05.964630 16096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:55:05.968369 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 16096
I20260929 15:55:05.968570 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:55:05.969103 388 external_mini_cluster.cc:1428] Setting key 5865432e66cad3bd914fd82c6a156390
Sep 29 15:55:05 dist-test-slave-r0ck krb5kdc[15856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:55:05.984551 16092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:05.967239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43743 (local address 127.0.97.62:38491)
0929 15:55:05.967459 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 15:55:05.967466 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:55:05.968558 (+ 1092us) server_negotiation.cc:407] Connection header received
0929 15:55:05.970635 (+ 2077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:55:05.970641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:55:05.970734 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:55:05.970841 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:55:05.972249 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:05.972942 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:05.974491 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:05.974924 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:05.975054 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:05.978651 (+ 3597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:55:05.978655 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:55:05.978659 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:55:05.978689 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:55:05.981180 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:55:05.982134 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:55:05.982147 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:55:05.982149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:55:05.982281 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:55:05.982818 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:55:05.982822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:55:05.982824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:55:05.983085 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:55:05.983234 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 15:55:05.984131 (+ 897us) server_negotiation.cc:299] Negotiation successful
0929 15:55:05.984305 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 15:55:05.985947 16226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:05.967656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.2:43743)
0929 15:55:05.968369 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:05.968432 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 15:55:05.970377 (+ 1945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:55:05.971059 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:55:05.971068 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:55:05.971560 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:55:05.972081 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:55:05.972096 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:55:05.973098 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:55:05.973102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:55:05.974342 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:55:05.974354 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:55:05.975147 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:55:05.975154 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:55:05.975371 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:05.976068 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:55:05.976089 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:55:05.978445 (+ 2356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:55:05.981323 (+ 2878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:55:05.981334 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:55:05.981376 (+ 42us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:55:05.981966 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:55:05.982395 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:55:05.982403 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:55:05.982409 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:55:05.982659 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:55:05.983234 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:55:05.983244 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:55:05.983680 (+ 436us) client_negotiation.cc:769] Sending connection context
0929 15:55:05.984007 (+ 327us) client_negotiation.cc:240] Negotiation successful
0929 15:55:05.984555 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":299,"threads_started":1}
I20260929 15:55:05.987382 16224 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38491
I20260929 15:55:05.987691 16224 heartbeater.cc:464] Registering TS with master...
I20260929 15:55:05.988411 16224 heartbeater.cc:511] Master 127.0.97.62:38491 requested a full tablet report, sending...
I20260929 15:55:05.989919 15901 ts_manager.cc:194] Registered new tserver with Master: b2cce6a0fc254095826871e8d5a0a0fc (127.0.97.2:37361)
I20260929 15:55:05.990756 15901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43743
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 15:55:06.025660 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38491
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:55:06.347015 16229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:55:06.347270 16229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:55:06.347360 16229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:55:06.354436 16229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:55:06.354533 16229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:55:06.354641 16229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:55:06.363926 16229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38491
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:55:06.365067 16229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:55:06.366313 16229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:55:06.381112 16234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:55:06.381117 16235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:55:06.381112 16237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:55:06.382041 16229 server_base.cc:1034] running on GCE node
I20260929 15:55:06.382642 16229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:55:06.383585 16229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:55:06.384835 16229 hybrid_clock.cc:648] HybridClock initialized: now 1790697306384789 us; error 57 us; skew 500 ppm
I20260929 15:55:06.385180 16229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:55:06 dist-test-slave-r0ck krb5kdc[15856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:55:06.388361 16229 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:55:06.389923 16229 webserver.cc:460] Webserver started at http://127.0.97.3:34951/ using document root <none> and password file <none>
I20260929 15:55:06.390581 16229 fs_manager.cc:359] Metadata directory not provided
I20260929 15:55:06.390727 16229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:55:06.391036 16229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:55:06.393944 16229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/data/instance:
uuid: "129d2804e3d54d01ad6eef1c52714dc9"
format_stamp: "Formatted at 2026-09-29 15:55:06 on dist-test-slave-r0ck"
server_key: "2305eba166af36f9aa4e4e048b2da4f6"
server_key_iv: "6546181ab497dccfbdc51021702bded9"
server_key_version: "encryptionkey@0"
I20260929 15:55:06.394692 16229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "129d2804e3d54d01ad6eef1c52714dc9"
format_stamp: "Formatted at 2026-09-29 15:55:06 on dist-test-slave-r0ck"
server_key: "2305eba166af36f9aa4e4e048b2da4f6"
server_key_iv: "6546181ab497dccfbdc51021702bded9"
server_key_version: "encryptionkey@0"
I20260929 15:55:06.399816 16229 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 15:55:06.403173 16244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:55:06.404444 16229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 15:55:06.404628 16229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/wal
uuid: "129d2804e3d54d01ad6eef1c52714dc9"
format_stamp: "Formatted at 2026-09-29 15:55:06 on dist-test-slave-r0ck"
server_key: "2305eba166af36f9aa4e4e048b2da4f6"
server_key_iv: "6546181ab497dccfbdc51021702bded9"
server_key_version: "encryptionkey@0"
I20260929 15:55:06.404839 16229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:55:06.429746 16229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:55:06.434280 16229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:55:06.434584 16229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:55:06.435573 16229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:55:06.436915 16229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:55:06.437022 16229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:55:06.437175 16229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:55:06.437247 16229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:55:06.471242 16229 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39479
I20260929 15:55:06.471333 16356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39479 every 8 connection(s)
I20260929 15:55:06.472539 16229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:55:06.473008 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 16229
I20260929 15:55:06.473210 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:55:06.473857 388 external_mini_cluster.cc:1428] Setting key 092fc18b4c851cd38064642ea1078edc
Sep 29 15:55:06 dist-test-slave-r0ck krb5kdc[15856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:55:06.493438 16092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:06.475268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41289 (local address 127.0.97.62:38491)
0929 15:55:06.475483 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 15:55:06.475489 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:55:06.476933 (+ 1444us) server_negotiation.cc:407] Connection header received
0929 15:55:06.479478 (+ 2545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:55:06.479482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:55:06.479547 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:55:06.479628 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:55:06.481551 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:06.482292 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:06.483785 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:06.484186 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:06.484319 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:06.487804 (+ 3485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:55:06.487809 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:55:06.487812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:55:06.487843 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:55:06.490876 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:55:06.491506 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:55:06.491510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:55:06.491512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:55:06.491644 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:55:06.491999 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:55:06.492001 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:55:06.492003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:55:06.492236 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:55:06.492342 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 15:55:06.492960 (+ 618us) server_negotiation.cc:299] Negotiation successful
0929 15:55:06.493108 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 15:55:06.494297 16359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:06.475913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.3:41289)
0929 15:55:06.476760 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:06.476813 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 15:55:06.479210 (+ 2397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:55:06.479892 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:55:06.479903 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:55:06.480616 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:55:06.481335 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:55:06.481405 (+ 70us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:55:06.482458 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:55:06.482463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:55:06.483641 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:55:06.483653 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:55:06.484407 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:55:06.484417 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:55:06.484649 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:06.485333 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:55:06.485451 (+ 118us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:55:06.487600 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:55:06.490994 (+ 3394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:55:06.491000 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:55:06.491011 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:55:06.491396 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:55:06.491732 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:55:06.491736 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:55:06.491738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:55:06.491902 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:55:06.492349 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:55:06.492355 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:55:06.492641 (+ 286us) client_negotiation.cc:769] Sending connection context
0929 15:55:06.492851 (+ 210us) client_negotiation.cc:240] Negotiation successful
0929 15:55:06.493213 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":331,"threads_started":1}
I20260929 15:55:06.495760 16357 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38491
I20260929 15:55:06.496075 16357 heartbeater.cc:464] Registering TS with master...
I20260929 15:55:06.496811 16357 heartbeater.cc:511] Master 127.0.97.62:38491 requested a full tablet report, sending...
I20260929 15:55:06.498221 15901 ts_manager.cc:194] Registered new tserver with Master: 129d2804e3d54d01ad6eef1c52714dc9 (127.0.97.3:39479)
I20260929 15:55:06.499069 15901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:41289
I20260929 15:55:06.501740 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:55:06.510222 16090 heartbeater.cc:503] Master 127.0.97.62:38491 was elected leader, sending a full tablet report...
I20260929 15:55:06.519750 16092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:06.506402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55512 (local address 127.0.97.62:38491)
0929 15:55:06.506627 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 15:55:06.506635 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:55:06.506953 (+ 318us) server_negotiation.cc:407] Connection header received
0929 15:55:06.507196 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:55:06.507201 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:55:06.507275 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:55:06.507381 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:55:06.508871 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:06.509702 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:06.511345 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:06.511976 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:06.512097 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:06.514510 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:55:06.514514 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:55:06.514518 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:55:06.514587 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:55:06.517216 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:55:06.517908 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:55:06.517912 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:55:06.517914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:55:06.518037 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:55:06.518451 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:55:06.518454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:55:06.518456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:55:06.518767 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:55:06.518966 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 15:55:06.519242 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 15:55:06.519405 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260929 15:55:06.523658 15901 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55512:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:55:06.530331 15901 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:55:06.563575 16370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:06.542290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39492 (local address 127.0.97.2:37361)
0929 15:55:06.555116 (+ 12826us) server_negotiation.cc:206] Beginning negotiation
0929 15:55:06.555128 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:55:06.555158 (+ 30us) server_negotiation.cc:407] Connection header received
0929 15:55:06.555269 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:55:06.555282 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:55:06.555541 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:55:06.555728 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:55:06.557257 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:06.558778 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:06.561407 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:06.562460 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:06.562610 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:06.562807 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 15:55:06.563092 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 15:55:06.563249 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12632,"thread_start_us":13617,"threads_started":1}
I20260929 15:55:06.563671 16371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:06.542141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37361 (local address 127.0.0.1:39492)
0929 15:55:06.553959 (+ 11818us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:06.554012 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 15:55:06.554243 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:55:06.556463 (+ 2220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:55:06.556471 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:55:06.556511 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:55:06.557037 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:55:06.557052 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:55:06.558922 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:55:06.558927 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:55:06.561208 (+ 2281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:55:06.561221 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:55:06.562673 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:55:06.562680 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:55:06.562873 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:06.562910 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 15:55:06.563043 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 15:55:06.563225 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11493,"mutex_wait_us":3,"thread_start_us":12300,"threads_started":1}
I20260929 15:55:06.564486 16369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:06.542385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35363 (local address 127.0.0.1:41202)
0929 15:55:06.554680 (+ 12295us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:06.554703 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 15:55:06.554846 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:55:06.555977 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:55:06.555984 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:55:06.556020 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:55:06.556457 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:55:06.556471 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:55:06.558457 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:55:06.558464 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:55:06.561605 (+ 3141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:55:06.561618 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:55:06.563477 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:55:06.563485 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:55:06.563671 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:06.563705 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 15:55:06.563771 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 15:55:06.563894 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12127,"mutex_wait_us":3,"thread_start_us":11086,"threads_started":1}
I20260929 15:55:06.565055 16372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:06.542792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41202 (local address 127.0.97.1:35363)
0929 15:55:06.555401 (+ 12609us) server_negotiation.cc:206] Beginning negotiation
0929 15:55:06.555409 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:55:06.555432 (+ 23us) server_negotiation.cc:407] Connection header received
0929 15:55:06.555515 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:55:06.555522 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:55:06.555730 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:55:06.556661 (+ 931us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:55:06.556828 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:06.558316 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:06.561781 (+ 3465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:06.563227 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:06.564306 (+ 1079us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:06.564533 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 15:55:06.564592 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 15:55:06.564735 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12455,"thread_start_us":12289,"threads_started":1}
I20260929 15:55:06.565565 16374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:06.555431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39479 (local address 127.0.0.1:41110)
0929 15:55:06.557649 (+ 2218us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:06.557665 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:55:06.557774 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:55:06.558500 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:55:06.558505 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:55:06.558533 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:55:06.559051 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:55:06.559064 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:55:06.560346 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:55:06.560353 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:55:06.562644 (+ 2291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:55:06.562663 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:55:06.564763 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:55:06.564772 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:55:06.564953 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:06.564984 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 15:55:06.565067 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 15:55:06.565180 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2087,"mutex_wait_us":27,"thread_start_us":1964,"threads_started":1}
I20260929 15:55:06.566298 16373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:06.555218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41110 (local address 127.0.97.3:39479)
0929 15:55:06.556549 (+ 1331us) server_negotiation.cc:206] Beginning negotiation
0929 15:55:06.556558 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:55:06.557903 (+ 1345us) server_negotiation.cc:407] Connection header received
0929 15:55:06.557997 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:55:06.558005 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:55:06.558207 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:55:06.558381 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:55:06.559231 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:06.560206 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:06.562856 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:06.564267 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:06.564387 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:06.564622 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 15:55:06.565781 (+ 1159us) server_negotiation.cc:299] Negotiation successful
0929 15:55:06.565961 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1153,"thread_start_us":988,"threads_started":1}
I20260929 15:55:06.567823 16159 tablet_service.cc:1467] Processing CreateTablet for tablet dec3e66df210465c852bee59b127c6d4 (DEFAULT_TABLE table=test-table [id=cca5ad2855d8453c9d25292b3547d49d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:55:06.568531 16025 tablet_service.cc:1467] Processing CreateTablet for tablet dec3e66df210465c852bee59b127c6d4 (DEFAULT_TABLE table=test-table [id=cca5ad2855d8453c9d25292b3547d49d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:55:06.569175 16159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dec3e66df210465c852bee59b127c6d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:55:06.569970 16025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dec3e66df210465c852bee59b127c6d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:55:06.569743 16292 tablet_service.cc:1467] Processing CreateTablet for tablet dec3e66df210465c852bee59b127c6d4 (DEFAULT_TABLE table=test-table [id=cca5ad2855d8453c9d25292b3547d49d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:55:06.571043 16292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dec3e66df210465c852bee59b127c6d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:55:06.580875 16375 tablet_bootstrap.cc:492] T dec3e66df210465c852bee59b127c6d4 P b2cce6a0fc254095826871e8d5a0a0fc: Bootstrap starting.
I20260929 15:55:06.583914 16375 tablet_bootstrap.cc:654] T dec3e66df210465c852bee59b127c6d4 P b2cce6a0fc254095826871e8d5a0a0fc: Neither blocks nor log segments found. Creating new log.
I20260929 15:55:06.584681 16376 tablet_bootstrap.cc:492] T dec3e66df210465c852bee59b127c6d4 P 725fc31fc8b241cb8990f3e8cfb8ac44: Bootstrap starting.
I20260929 15:55:06.585196 16375 log.cc:824] T dec3e66df210465c852bee59b127c6d4 P b2cce6a0fc254095826871e8d5a0a0fc: Log is configured to *not* fsync() on all Append() calls
I20260929 15:55:06.586944 16377 tablet_bootstrap.cc:492] T dec3e66df210465c852bee59b127c6d4 P 129d2804e3d54d01ad6eef1c52714dc9: Bootstrap starting.
I20260929 15:55:06.588205 16375 tablet_bootstrap.cc:492] T dec3e66df210465c852bee59b127c6d4 P b2cce6a0fc254095826871e8d5a0a0fc: No bootstrap required, opened a new log
I20260929 15:55:06.588502 16375 ts_tablet_manager.cc:1397] T dec3e66df210465c852bee59b127c6d4 P b2cce6a0fc254095826871e8d5a0a0fc: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 15:55:06.589466 16376 tablet_bootstrap.cc:654] T dec3e66df210465c852bee59b127c6d4 P 725fc31fc8b241cb8990f3e8cfb8ac44: Neither blocks nor log segments found. Creating new log.
I20260929 15:55:06.590273 16377 tablet_bootstrap.cc:654] T dec3e66df210465c852bee59b127c6d4 P 129d2804e3d54d01ad6eef1c52714dc9: Neither blocks nor log segments found. Creating new log.
I20260929 15:55:06.590692 16376 log.cc:824] T dec3e66df210465c852bee59b127c6d4 P 725fc31fc8b241cb8990f3e8cfb8ac44: Log is configured to *not* fsync() on all Append() calls
I20260929 15:55:06.591459 16377 log.cc:824] T dec3e66df210465c852bee59b127c6d4 P 129d2804e3d54d01ad6eef1c52714dc9: Log is configured to *not* fsync() on all Append() calls
I20260929 15:55:06.593530 16376 tablet_bootstrap.cc:492] T dec3e66df210465c852bee59b127c6d4 P 725fc31fc8b241cb8990f3e8cfb8ac44: No bootstrap required, opened a new log
I20260929 15:55:06.593780 16376 ts_tablet_manager.cc:1397] T dec3e66df210465c852bee59b127c6d4 P 725fc31fc8b241cb8990f3e8cfb8ac44: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 15:55:06.593804 16377 tablet_bootstrap.cc:492] T dec3e66df210465c852bee59b127c6d4 P 129d2804e3d54d01ad6eef1c52714dc9: No bootstrap required, opened a new log
I20260929 15:55:06.594053 16377 ts_tablet_manager.cc:1397] T dec3e66df210465c852bee59b127c6d4 P 129d2804e3d54d01ad6eef1c52714dc9: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 15:55:06.598237 16375 raft_consensus.cc:348] T dec3e66df210465c852bee59b127c6d4 P b2cce6a0fc254095826871e8d5a0a0fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2cce6a0fc254095826871e8d5a0a0fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37361 } } peers { permanent_uuid: "725fc31fc8b241cb8990f3e8cfb8ac44" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35363 } } peers { permanent_uuid: "129d2804e3d54d01ad6eef1c52714dc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39479 } }
I20260929 15:55:06.598671 16375 raft_consensus.cc:374] T dec3e66df210465c852bee59b127c6d4 P b2cce6a0fc254095826871e8d5a0a0fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:55:06.598826 16375 raft_consensus.cc:729] T dec3e66df210465c852bee59b127c6d4 P b2cce6a0fc254095826871e8d5a0a0fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2cce6a0fc254095826871e8d5a0a0fc, State: Initialized, Role: FOLLOWER
I20260929 15:55:06.599534 16375 consensus_queue.cc:260] T dec3e66df210465c852bee59b127c6d4 P b2cce6a0fc254095826871e8d5a0a0fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2cce6a0fc254095826871e8d5a0a0fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37361 } } peers { permanent_uuid: "725fc31fc8b241cb8990f3e8cfb8ac44" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35363 } } peers { permanent_uuid: "129d2804e3d54d01ad6eef1c52714dc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39479 } }
I20260929 15:55:06.600642 16224 heartbeater.cc:503] Master 127.0.97.62:38491 was elected leader, sending a full tablet report...
I20260929 15:55:06.601315 16375 ts_tablet_manager.cc:1428] T dec3e66df210465c852bee59b127c6d4 P b2cce6a0fc254095826871e8d5a0a0fc: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260929 15:55:06.602420 16377 raft_consensus.cc:348] T dec3e66df210465c852bee59b127c6d4 P 129d2804e3d54d01ad6eef1c52714dc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2cce6a0fc254095826871e8d5a0a0fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37361 } } peers { permanent_uuid: "725fc31fc8b241cb8990f3e8cfb8ac44" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35363 } } peers { permanent_uuid: "129d2804e3d54d01ad6eef1c52714dc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39479 } }
I20260929 15:55:06.602798 16377 raft_consensus.cc:374] T dec3e66df210465c852bee59b127c6d4 P 129d2804e3d54d01ad6eef1c52714dc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:55:06.602902 16377 raft_consensus.cc:729] T dec3e66df210465c852bee59b127c6d4 P 129d2804e3d54d01ad6eef1c52714dc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129d2804e3d54d01ad6eef1c52714dc9, State: Initialized, Role: FOLLOWER
I20260929 15:55:06.603488 16377 consensus_queue.cc:260] T dec3e66df210465c852bee59b127c6d4 P 129d2804e3d54d01ad6eef1c52714dc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2cce6a0fc254095826871e8d5a0a0fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37361 } } peers { permanent_uuid: "725fc31fc8b241cb8990f3e8cfb8ac44" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35363 } } peers { permanent_uuid: "129d2804e3d54d01ad6eef1c52714dc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39479 } }
I20260929 15:55:06.604149 16376 raft_consensus.cc:348] T dec3e66df210465c852bee59b127c6d4 P 725fc31fc8b241cb8990f3e8cfb8ac44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2cce6a0fc254095826871e8d5a0a0fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37361 } } peers { permanent_uuid: "725fc31fc8b241cb8990f3e8cfb8ac44" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35363 } } peers { permanent_uuid: "129d2804e3d54d01ad6eef1c52714dc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39479 } }
I20260929 15:55:06.604550 16357 heartbeater.cc:503] Master 127.0.97.62:38491 was elected leader, sending a full tablet report...
I20260929 15:55:06.604557 16376 raft_consensus.cc:374] T dec3e66df210465c852bee59b127c6d4 P 725fc31fc8b241cb8990f3e8cfb8ac44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:55:06.604674 16376 raft_consensus.cc:729] T dec3e66df210465c852bee59b127c6d4 P 725fc31fc8b241cb8990f3e8cfb8ac44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 725fc31fc8b241cb8990f3e8cfb8ac44, State: Initialized, Role: FOLLOWER
I20260929 15:55:06.605187 16376 consensus_queue.cc:260] T dec3e66df210465c852bee59b127c6d4 P 725fc31fc8b241cb8990f3e8cfb8ac44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2cce6a0fc254095826871e8d5a0a0fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37361 } } peers { permanent_uuid: "725fc31fc8b241cb8990f3e8cfb8ac44" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35363 } } peers { permanent_uuid: "129d2804e3d54d01ad6eef1c52714dc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39479 } }
I20260929 15:55:06.606973 16376 ts_tablet_manager.cc:1428] T dec3e66df210465c852bee59b127c6d4 P 725fc31fc8b241cb8990f3e8cfb8ac44: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260929 15:55:06.607914 16377 ts_tablet_manager.cc:1428] T dec3e66df210465c852bee59b127c6d4 P 129d2804e3d54d01ad6eef1c52714dc9: Time spent starting tablet: real 0.014s user 0.011s sys 0.000s
I20260929 15:55:06.668926 16383 raft_consensus.cc:482] T dec3e66df210465c852bee59b127c6d4 P 725fc31fc8b241cb8990f3e8cfb8ac44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:55:06.669209 16383 raft_consensus.cc:504] T dec3e66df210465c852bee59b127c6d4 P 725fc31fc8b241cb8990f3e8cfb8ac44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2cce6a0fc254095826871e8d5a0a0fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37361 } } peers { permanent_uuid: "725fc31fc8b241cb8990f3e8cfb8ac44" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35363 } } peers { permanent_uuid: "129d2804e3d54d01ad6eef1c52714dc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39479 } }
I20260929 15:55:06.670879 16383 leader_election.cc:288] T dec3e66df210465c852bee59b127c6d4 P 725fc31fc8b241cb8990f3e8cfb8ac44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2cce6a0fc254095826871e8d5a0a0fc (127.0.97.2:37361), 129d2804e3d54d01ad6eef1c52714dc9 (127.0.97.3:39479)
I20260929 15:55:06.679776 16373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:06.672613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45063 (local address 127.0.97.3:39479)
0929 15:55:06.672819 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 15:55:06.672824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:55:06.672985 (+ 161us) server_negotiation.cc:407] Connection header received
0929 15:55:06.673237 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:55:06.673247 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:55:06.673306 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:55:06.673439 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:55:06.673916 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:06.674895 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:06.677259 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:06.678515 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:06.678631 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:06.678731 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 15:55:06.679387 (+ 656us) server_negotiation.cc:299] Negotiation successful
0929 15:55:06.679530 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 15:55:06.679932 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:06.671452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37361 (local address 127.0.97.1:47423)
0929 15:55:06.672108 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:06.672131 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 15:55:06.672286 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:55:06.672889 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:55:06.672892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:55:06.672919 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:55:06.673278 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:55:06.673286 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:55:06.674693 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:55:06.674697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:55:06.677006 (+ 2309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:55:06.677024 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:55:06.678855 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:55:06.678860 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:55:06.679157 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:06.679235 (+ 78us) client_negotiation.cc:769] Sending connection context
0929 15:55:06.679331 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 15:55:06.679515 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":119,"spinlock_wait_cycles":29568,"thread_start_us":330,"threads_started":1}
I20260929 15:55:06.680330 16385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:06.672259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39479 (local address 127.0.97.1:45063)
0929 15:55:06.672865 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:06.672895 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 15:55:06.673090 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:55:06.673459 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:55:06.673463 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:55:06.673496 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:55:06.673765 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:55:06.673772 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:55:06.675045 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:55:06.675051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:55:06.677106 (+ 2055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:55:06.677120 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:55:06.678832 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:55:06.678844 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:55:06.679156 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:06.679227 (+ 71us) client_negotiation.cc:769] Sending connection context
0929 15:55:06.679331 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 15:55:06.679960 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":499,"thread_start_us":283,"threads_started":1}
I20260929 15:55:06.679776 16370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:06.671642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47423 (local address 127.0.97.2:37361)
0929 15:55:06.671860 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 15:55:06.671864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:55:06.672252 (+ 388us) server_negotiation.cc:407] Connection header received
0929 15:55:06.672514 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:55:06.672523 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:55:06.672617 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:55:06.672686 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:55:06.673451 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:06.674521 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:06.677216 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:06.678479 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:06.678631 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:06.678729 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 15:55:06.679370 (+ 641us) server_negotiation.cc:299] Negotiation successful
0929 15:55:06.679505 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 15:55:06.681330 16312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dec3e66df210465c852bee59b127c6d4" candidate_uuid: "725fc31fc8b241cb8990f3e8cfb8ac44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "129d2804e3d54d01ad6eef1c52714dc9" is_pre_election: true
I20260929 15:55:06.681628 16179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dec3e66df210465c852bee59b127c6d4" candidate_uuid: "725fc31fc8b241cb8990f3e8cfb8ac44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2cce6a0fc254095826871e8d5a0a0fc" is_pre_election: true
I20260929 15:55:06.682005 16312 raft_consensus.cc:2454] T dec3e66df210465c852bee59b127c6d4 P 129d2804e3d54d01ad6eef1c52714dc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 725fc31fc8b241cb8990f3e8cfb8ac44 in term 0.
I20260929 15:55:06.682190 16179 raft_consensus.cc:2454] T dec3e66df210465c852bee59b127c6d4 P b2cce6a0fc254095826871e8d5a0a0fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 725fc31fc8b241cb8990f3e8cfb8ac44 in term 0.
I20260929 15:55:06.682705 15979 leader_election.cc:302] T dec3e66df210465c852bee59b127c6d4 P 725fc31fc8b241cb8990f3e8cfb8ac44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 129d2804e3d54d01ad6eef1c52714dc9, 725fc31fc8b241cb8990f3e8cfb8ac44; no voters:
I20260929 15:55:06.683100 16383 raft_consensus.cc:2781] T dec3e66df210465c852bee59b127c6d4 P 725fc31fc8b241cb8990f3e8cfb8ac44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:55:06.683254 16383 raft_consensus.cc:482] T dec3e66df210465c852bee59b127c6d4 P 725fc31fc8b241cb8990f3e8cfb8ac44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:55:06.683370 16383 raft_consensus.cc:3037] T dec3e66df210465c852bee59b127c6d4 P 725fc31fc8b241cb8990f3e8cfb8ac44 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:55:06.685618 16383 raft_consensus.cc:504] T dec3e66df210465c852bee59b127c6d4 P 725fc31fc8b241cb8990f3e8cfb8ac44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2cce6a0fc254095826871e8d5a0a0fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37361 } } peers { permanent_uuid: "725fc31fc8b241cb8990f3e8cfb8ac44" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35363 } } peers { permanent_uuid: "129d2804e3d54d01ad6eef1c52714dc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39479 } }
I20260929 15:55:06.686215 16383 leader_election.cc:288] T dec3e66df210465c852bee59b127c6d4 P 725fc31fc8b241cb8990f3e8cfb8ac44 [CANDIDATE]: Term 1 election: Requested vote from peers b2cce6a0fc254095826871e8d5a0a0fc (127.0.97.2:37361), 129d2804e3d54d01ad6eef1c52714dc9 (127.0.97.3:39479)
I20260929 15:55:06.686635 16179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dec3e66df210465c852bee59b127c6d4" candidate_uuid: "725fc31fc8b241cb8990f3e8cfb8ac44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2cce6a0fc254095826871e8d5a0a0fc"
I20260929 15:55:06.686810 16179 raft_consensus.cc:3037] T dec3e66df210465c852bee59b127c6d4 P b2cce6a0fc254095826871e8d5a0a0fc [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:55:06.686816 16312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dec3e66df210465c852bee59b127c6d4" candidate_uuid: "725fc31fc8b241cb8990f3e8cfb8ac44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "129d2804e3d54d01ad6eef1c52714dc9"
I20260929 15:55:06.687040 16312 raft_consensus.cc:3037] T dec3e66df210465c852bee59b127c6d4 P 129d2804e3d54d01ad6eef1c52714dc9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:55:06.688951 16179 raft_consensus.cc:2454] T dec3e66df210465c852bee59b127c6d4 P b2cce6a0fc254095826871e8d5a0a0fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 725fc31fc8b241cb8990f3e8cfb8ac44 in term 1.
I20260929 15:55:06.689280 16312 raft_consensus.cc:2454] T dec3e66df210465c852bee59b127c6d4 P 129d2804e3d54d01ad6eef1c52714dc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 725fc31fc8b241cb8990f3e8cfb8ac44 in term 1.
I20260929 15:55:06.689311 15979 leader_election.cc:302] T dec3e66df210465c852bee59b127c6d4 P 725fc31fc8b241cb8990f3e8cfb8ac44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 725fc31fc8b241cb8990f3e8cfb8ac44, b2cce6a0fc254095826871e8d5a0a0fc; no voters:
I20260929 15:55:06.689683 16383 raft_consensus.cc:2781] T dec3e66df210465c852bee59b127c6d4 P 725fc31fc8b241cb8990f3e8cfb8ac44 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:55:06.690088 16383 raft_consensus.cc:686] T dec3e66df210465c852bee59b127c6d4 P 725fc31fc8b241cb8990f3e8cfb8ac44 [term 1 LEADER]: Becoming Leader. State: Replica: 725fc31fc8b241cb8990f3e8cfb8ac44, State: Running, Role: LEADER
I20260929 15:55:06.690532 16383 consensus_queue.cc:237] T dec3e66df210465c852bee59b127c6d4 P 725fc31fc8b241cb8990f3e8cfb8ac44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2cce6a0fc254095826871e8d5a0a0fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37361 } } peers { permanent_uuid: "725fc31fc8b241cb8990f3e8cfb8ac44" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35363 } } peers { permanent_uuid: "129d2804e3d54d01ad6eef1c52714dc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39479 } }
I20260929 15:55:06.695024 15901 catalog_manager.cc:5495] T dec3e66df210465c852bee59b127c6d4 P 725fc31fc8b241cb8990f3e8cfb8ac44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 725fc31fc8b241cb8990f3e8cfb8ac44 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "725fc31fc8b241cb8990f3e8cfb8ac44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2cce6a0fc254095826871e8d5a0a0fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "725fc31fc8b241cb8990f3e8cfb8ac44" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "129d2804e3d54d01ad6eef1c52714dc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39479 } health_report { overall_health: UNKNOWN } } }
W20260929 15:55:06.716691 16225 tablet.cc:2369] T dec3e66df210465c852bee59b127c6d4 P b2cce6a0fc254095826871e8d5a0a0fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:55:06.721925 16372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:06.716944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41216 (local address 127.0.97.1:35363)
0929 15:55:06.717129 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 15:55:06.717133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:55:06.717149 (+ 16us) server_negotiation.cc:407] Connection header received
0929 15:55:06.717288 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:55:06.717291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:55:06.717430 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:55:06.717568 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:55:06.717923 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:06.718621 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:06.719923 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:06.720292 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:06.720378 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:06.720790 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:55:06.721254 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:55:06.721424 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 15:55:06.721573 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 15:55:06.721635 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260929 15:55:06.724223 16358 tablet.cc:2369] T dec3e66df210465c852bee59b127c6d4 P 129d2804e3d54d01ad6eef1c52714dc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:55:06.728855 16091 tablet.cc:2369] T dec3e66df210465c852bee59b127c6d4 P 725fc31fc8b241cb8990f3e8cfb8ac44: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:55:06.729976 16312 raft_consensus.cc:1261] T dec3e66df210465c852bee59b127c6d4 P 129d2804e3d54d01ad6eef1c52714dc9 [term 1 FOLLOWER]: Refusing update from remote peer 725fc31fc8b241cb8990f3e8cfb8ac44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:55:06.729976 16179 raft_consensus.cc:1261] T dec3e66df210465c852bee59b127c6d4 P b2cce6a0fc254095826871e8d5a0a0fc [term 1 FOLLOWER]: Refusing update from remote peer 725fc31fc8b241cb8990f3e8cfb8ac44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:55:06.730887 16383 consensus_queue.cc:1035] T dec3e66df210465c852bee59b127c6d4 P 725fc31fc8b241cb8990f3e8cfb8ac44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129d2804e3d54d01ad6eef1c52714dc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:55:06.731199 16386 consensus_queue.cc:1035] T dec3e66df210465c852bee59b127c6d4 P 725fc31fc8b241cb8990f3e8cfb8ac44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2cce6a0fc254095826871e8d5a0a0fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:55:06.754766 15901 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:55:06.754980 15901 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:55:06.758471 15901 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9: Sending DeleteTablet for 3 replicas of tablet dec3e66df210465c852bee59b127c6d4
I20260929 15:55:06.759618 16025 tablet_service.cc:1514] Processing DeleteTablet for tablet dec3e66df210465c852bee59b127c6d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:55:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41202
I20260929 15:55:06.759618 16159 tablet_service.cc:1514] Processing DeleteTablet for tablet dec3e66df210465c852bee59b127c6d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:55:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39492
I20260929 15:55:06.759740 16292 tablet_service.cc:1514] Processing DeleteTablet for tablet dec3e66df210465c852bee59b127c6d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:55:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41110
I20260929 15:55:06.760792 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 15962
I20260929 15:55:06.772060 16400 tablet_replica.cc:330] T dec3e66df210465c852bee59b127c6d4 P 725fc31fc8b241cb8990f3e8cfb8ac44: stopping tablet replica
I20260929 15:55:06.772732 16399 tablet_replica.cc:330] T dec3e66df210465c852bee59b127c6d4 P b2cce6a0fc254095826871e8d5a0a0fc: stopping tablet replica
I20260929 15:55:06.773495 16399 raft_consensus.cc:2229] T dec3e66df210465c852bee59b127c6d4 P b2cce6a0fc254095826871e8d5a0a0fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:55:06.774120 16400 raft_consensus.cc:2229] T dec3e66df210465c852bee59b127c6d4 P 725fc31fc8b241cb8990f3e8cfb8ac44 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:55:06.774973 16400 raft_consensus.cc:2258] T dec3e66df210465c852bee59b127c6d4 P 725fc31fc8b241cb8990f3e8cfb8ac44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:55:06.775621 16399 raft_consensus.cc:2258] T dec3e66df210465c852bee59b127c6d4 P b2cce6a0fc254095826871e8d5a0a0fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:55:06.778530 16400 ts_tablet_manager.cc:1905] T dec3e66df210465c852bee59b127c6d4 P 725fc31fc8b241cb8990f3e8cfb8ac44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:55:06.780150 16399 ts_tablet_manager.cc:1905] T dec3e66df210465c852bee59b127c6d4 P b2cce6a0fc254095826871e8d5a0a0fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:55:06.784523 16401 tablet_replica.cc:330] T dec3e66df210465c852bee59b127c6d4 P 129d2804e3d54d01ad6eef1c52714dc9: stopping tablet replica
I20260929 15:55:06.784920 16401 raft_consensus.cc:2229] T dec3e66df210465c852bee59b127c6d4 P 129d2804e3d54d01ad6eef1c52714dc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:55:06.785480 16401 raft_consensus.cc:2258] T dec3e66df210465c852bee59b127c6d4 P 129d2804e3d54d01ad6eef1c52714dc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:55:06.787124 16399 ts_tablet_manager.cc:1918] T dec3e66df210465c852bee59b127c6d4 P b2cce6a0fc254095826871e8d5a0a0fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:55:06.787470 16401 ts_tablet_manager.cc:1905] T dec3e66df210465c852bee59b127c6d4 P 129d2804e3d54d01ad6eef1c52714dc9: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 15:55:06 dist-test-slave-r0ck krb5kdc[15856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 15:55:06.790753 16399 log.cc:1196] T dec3e66df210465c852bee59b127c6d4 P b2cce6a0fc254095826871e8d5a0a0fc: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/wal/wals/dec3e66df210465c852bee59b127c6d4
I20260929 15:55:06.794442 16401 ts_tablet_manager.cc:1918] T dec3e66df210465c852bee59b127c6d4 P 129d2804e3d54d01ad6eef1c52714dc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:55:06.794687 16401 log.cc:1196] T dec3e66df210465c852bee59b127c6d4 P 129d2804e3d54d01ad6eef1c52714dc9: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/wal/wals/dec3e66df210465c852bee59b127c6d4
I20260929 15:55:06.794747 16400 ts_tablet_manager.cc:1918] T dec3e66df210465c852bee59b127c6d4 P 725fc31fc8b241cb8990f3e8cfb8ac44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:55:06.795006 16400 log.cc:1196] T dec3e66df210465c852bee59b127c6d4 P 725fc31fc8b241cb8990f3e8cfb8ac44: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/wal/wals/dec3e66df210465c852bee59b127c6d4
I20260929 15:55:06.795326 16401 ts_tablet_manager.cc:1939] T dec3e66df210465c852bee59b127c6d4 P 129d2804e3d54d01ad6eef1c52714dc9: Deleting consensus metadata
I20260929 15:55:06.795548 16400 ts_tablet_manager.cc:1939] T dec3e66df210465c852bee59b127c6d4 P 725fc31fc8b241cb8990f3e8cfb8ac44: Deleting consensus metadata
I20260929 15:55:06.797250 15886 catalog_manager.cc:4841] TS 129d2804e3d54d01ad6eef1c52714dc9 (127.0.97.3:39479): tablet dec3e66df210465c852bee59b127c6d4 (table test-table [id=cca5ad2855d8453c9d25292b3547d49d]) successfully deleted
I20260929 15:55:06.797392 15888 catalog_manager.cc:4841] TS 725fc31fc8b241cb8990f3e8cfb8ac44 (127.0.97.1:35363): tablet dec3e66df210465c852bee59b127c6d4 (table test-table [id=cca5ad2855d8453c9d25292b3547d49d]) successfully deleted
I20260929 15:55:06.798162 16372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:06.763517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41220 (local address 127.0.97.1:35363)
0929 15:55:06.763728 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 15:55:06.763737 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:55:06.776606 (+ 12869us) server_negotiation.cc:407] Connection header received
0929 15:55:06.777065 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:55:06.777072 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:55:06.777155 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:55:06.777235 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:55:06.779314 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:06.780429 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:06.782434 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:06.783149 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:06.783239 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:06.789694 (+ 6455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:55:06.789708 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:55:06.789724 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:55:06.789770 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:55:06.794023 (+ 4253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:55:06.795031 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:55:06.795039 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:55:06.795043 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:55:06.795267 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:55:06.795726 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:55:06.795731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:55:06.795735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:55:06.796337 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:55:06.796630 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0929 15:55:06.797552 (+ 922us) server_negotiation.cc:299] Negotiation successful
0929 15:55:06.797789 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 15:55:06.798738 16399 ts_tablet_manager.cc:1939] T dec3e66df210465c852bee59b127c6d4 P b2cce6a0fc254095826871e8d5a0a0fc: Deleting consensus metadata
I20260929 15:55:06.799315 16085 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:55:06.800308 15886 catalog_manager.cc:4841] TS b2cce6a0fc254095826871e8d5a0a0fc (127.0.97.2:37361): tablet dec3e66df210465c852bee59b127c6d4 (table test-table [id=cca5ad2855d8453c9d25292b3547d49d]) successfully deleted
I20260929 15:55:06.897300 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 15962
I20260929 15:55:06.916527 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 16096
Sep 29 15:55:06 dist-test-slave-r0ck krb5kdc[15856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 15:55:06.932217 16370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:06.917254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39496 (local address 127.0.97.2:37361)
0929 15:55:06.917459 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 15:55:06.917467 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:55:06.917555 (+ 88us) server_negotiation.cc:407] Connection header received
0929 15:55:06.917768 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:55:06.917773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:55:06.917851 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:55:06.917959 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:55:06.919389 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:06.920131 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:06.921920 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:06.922320 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:06.922460 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:06.926719 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:55:06.926742 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:55:06.926758 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:55:06.926795 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:55:06.929450 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:55:06.930106 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:55:06.930122 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:55:06.930126 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:55:06.930252 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:55:06.930664 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:55:06.930669 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:55:06.930672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:55:06.931041 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:55:06.931330 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0929 15:55:06.931650 (+ 320us) server_negotiation.cc:299] Negotiation successful
0929 15:55:06.931863 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 15:55:06.932878 16219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:55:07.022697 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 16096
I20260929 15:55:07.039428 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 16229
Sep 29 15:55:07 dist-test-slave-r0ck krb5kdc[15856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 15:55:07.054756 16373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:07.040180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41114 (local address 127.0.97.3:39479)
0929 15:55:07.040370 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 15:55:07.040378 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:55:07.040399 (+ 21us) server_negotiation.cc:407] Connection header received
0929 15:55:07.040638 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:55:07.040643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:55:07.040709 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:55:07.040830 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:55:07.042201 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:07.042942 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:07.044283 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:07.044701 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:07.044834 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:07.049187 (+ 4353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:55:07.049201 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:55:07.049218 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:55:07.049252 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:55:07.051830 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:55:07.052495 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:55:07.052509 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:55:07.052511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:55:07.052668 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:55:07.053046 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:55:07.053052 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:55:07.053055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:55:07.053598 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:55:07.053888 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0929 15:55:07.054193 (+ 305us) server_negotiation.cc:299] Negotiation successful
0929 15:55:07.054412 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 15:55:07.055405 16352 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:55:07.143266 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 16229
I20260929 15:55:07.160754 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 15869
I20260929 15:55:07.162123 15931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:55:07.271714 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 15869
I20260929 15:55:27.292881 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38491
--webserver_interface=127.0.97.62
--webserver_port=34989
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:55:27.617242 16407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:55:27.617590 16407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:55:27.617738 16407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:55:27.625488 16407 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 15:55:27.625578 16407 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 15:55:27.625684 16407 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 15:55:27.625779 16407 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:55:27.625878 16407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:55:27.625978 16407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:55:27.626057 16407 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:55:27.626139 16407 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:55:27.635825 16407 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=34989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:55:27.637070 16407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:55:27.638342 16407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:55:27.660898 16413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:55:27.660837 16414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:55:27.660852 16416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:55:27.666730 16407 server_base.cc:1034] running on GCE node
I20260929 15:55:27.667682 16407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:55:27.669278 16407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:55:27.670545 16407 hybrid_clock.cc:648] HybridClock initialized: now 1790697327670485 us; error 37 us; skew 500 ppm
I20260929 15:55:27.670930 16407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:55:27 dist-test-slave-r0ck krb5kdc[15856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:55:27.677829 16407 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:55:27.679610 16407 webserver.cc:460] Webserver started at http://127.0.97.62:34989/ using document root <none> and password file <none>
I20260929 15:55:27.680442 16407 fs_manager.cc:359] Metadata directory not provided
I20260929 15:55:27.680593 16407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:55:27.686569 16407 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 15:55:27.689553 16423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:55:27.690919 16407 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 15:55:27.691115 16407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "29da14e05d2b468ba3c91329fd95aff9"
format_stamp: "Formatted at 2026-09-29 15:55:04 on dist-test-slave-r0ck"
server_key: "f500f5b7e97e070a6e58e107e4e04045"
server_key_iv: "3bbd59075cf52066af916acb4c690389"
server_key_version: "encryptionkey@0"
I20260929 15:55:27.692152 16407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:55:27.707427 16407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:55:27.711544 16407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:55:27.711836 16407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:55:27.731067 16407 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38491
I20260929 15:55:27.731096 16474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38491 every 8 connection(s)
I20260929 15:55:27.732627 16407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:55:27.735615 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 16407
I20260929 15:55:27.736652 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:35363
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:38491
--webserver_port=36367
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:55:27.741938 16475 sys_catalog.cc:263] Verifying existing consensus state
I20260929 15:55:27.746757 16475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9: Bootstrap starting.
I20260929 15:55:27.767055 16475 log.cc:824] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9: Log is configured to *not* fsync() on all Append() calls
I20260929 15:55:27.779114 16475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 15:55:27.779682 16475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9: Bootstrap complete.
I20260929 15:55:27.788978 16475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29da14e05d2b468ba3c91329fd95aff9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38491 } }
I20260929 15:55:27.789862 16475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29da14e05d2b468ba3c91329fd95aff9, State: Initialized, Role: FOLLOWER
I20260929 15:55:27.790446 16475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29da14e05d2b468ba3c91329fd95aff9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38491 } }
I20260929 15:55:27.790707 16475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:55:27.790848 16475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:55:27.791026 16475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [term 1 FOLLOWER]: Advancing to term 2
I20260929 15:55:27.794792 16475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29da14e05d2b468ba3c91329fd95aff9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38491 } }
I20260929 15:55:27.795334 16475 leader_election.cc:302] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29da14e05d2b468ba3c91329fd95aff9; no voters:
I20260929 15:55:27.796047 16475 leader_election.cc:288] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 15:55:27.796182 16479 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 15:55:27.797487 16479 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [term 2 LEADER]: Becoming Leader. State: Replica: 29da14e05d2b468ba3c91329fd95aff9, State: Running, Role: LEADER
I20260929 15:55:27.798079 16479 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29da14e05d2b468ba3c91329fd95aff9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38491 } }
I20260929 15:55:27.798403 16475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:55:27.800455 16481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29da14e05d2b468ba3c91329fd95aff9. Latest consensus state: current_term: 2 leader_uuid: "29da14e05d2b468ba3c91329fd95aff9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29da14e05d2b468ba3c91329fd95aff9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38491 } } }
I20260929 15:55:27.800809 16481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [sys.catalog]: This master's current role is: LEADER
I20260929 15:55:27.800580 16480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "29da14e05d2b468ba3c91329fd95aff9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29da14e05d2b468ba3c91329fd95aff9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38491 } } }
I20260929 15:55:27.801002 16480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [sys.catalog]: This master's current role is: LEADER
I20260929 15:55:27.806496 16487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:55:27.823531 16487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:55:27.830907 16487 catalog_manager.cc:1259] Loaded cluster ID: b75c332d7dba4033b10c80d75da9bcb8
I20260929 15:55:27.831027 16487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:55:27.835045 16487 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:55:27.836539 16487 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9: Loaded TSK: 0
I20260929 15:55:27.875551 16487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9: Generated new TSK 1
I20260929 15:55:27.875967 16487 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 15:55:28.083762 16477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:55:28.084049 16477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:55:28.084210 16477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:55:28.091410 16477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:55:28.091562 16477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:55:28.091739 16477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:55:28.101665 16477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:35363
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=36367
--tserver_master_addrs=127.0.97.62:38491
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:55:28.102922 16477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:55:28.104314 16477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:55:28.118853 16502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:55:28.118845 16503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:55:28.119994 16505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:55:28.120311 16477 server_base.cc:1034] running on GCE node
I20260929 15:55:28.120913 16477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:55:28.121989 16477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:55:28.123206 16477 hybrid_clock.cc:648] HybridClock initialized: now 1790697328123153 us; error 46 us; skew 500 ppm
I20260929 15:55:28.123529 16477 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:55:28 dist-test-slave-r0ck krb5kdc[15856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:55:28.126838 16477 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:55:28.128422 16477 webserver.cc:460] Webserver started at http://127.0.97.1:36367/ using document root <none> and password file <none>
I20260929 15:55:28.129096 16477 fs_manager.cc:359] Metadata directory not provided
I20260929 15:55:28.129252 16477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:55:28.134509 16477 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:55:28.137177 16512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:55:28.138604 16477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 15:55:28.138763 16477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/wal
uuid: "725fc31fc8b241cb8990f3e8cfb8ac44"
format_stamp: "Formatted at 2026-09-29 15:55:05 on dist-test-slave-r0ck"
server_key: "51807be11621fed588914153c42c4e30"
server_key_iv: "9bb144e888d933f0b4c26c8ce5790384"
server_key_version: "encryptionkey@0"
I20260929 15:55:28.139945 16477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:55:28.167364 16477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:55:28.171723 16477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:55:28.172017 16477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:55:28.173128 16477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:55:28.174633 16477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:55:28.174750 16477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:55:28.174880 16477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:55:28.174980 16477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:55:28.209892 16477 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35363
I20260929 15:55:28.210004 16624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35363 every 8 connection(s)
I20260929 15:55:28.211378 16477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:55:28.212944 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 16477
I20260929 15:55:28.213706 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:37361
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:38491
--webserver_port=33541
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 15:55:28 dist-test-slave-r0ck krb5kdc[15856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:55:28.240675 16627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:28.214600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34677 (local address 127.0.97.62:38491)
0929 15:55:28.215364 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0929 15:55:28.215384 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 15:55:28.218166 (+ 2782us) server_negotiation.cc:407] Connection header received
0929 15:55:28.220383 (+ 2217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:55:28.220394 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:55:28.220776 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:55:28.221194 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:55:28.222691 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:28.224722 (+ 2031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:28.226326 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:28.227406 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:28.227591 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:28.231061 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:55:28.231074 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:55:28.231096 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:55:28.231147 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:55:28.235574 (+ 4427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:55:28.236634 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:55:28.236641 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:55:28.236646 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:55:28.236812 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:55:28.237430 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:55:28.237436 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:55:28.237439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:55:28.237921 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:55:28.238083 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 15:55:28.239054 (+ 971us) server_negotiation.cc:299] Negotiation successful
0929 15:55:28.239555 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":278,"threads_started":1}
I20260929 15:55:28.240818 16628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:28.214772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.1:34677)
0929 15:55:28.217966 (+ 3194us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:28.218024 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 15:55:28.219851 (+ 1827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:55:28.221418 (+ 1567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:55:28.221426 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:55:28.221966 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:55:28.222478 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:55:28.222496 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:55:28.224907 (+ 2411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:55:28.224912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:55:28.226180 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:55:28.226193 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:55:28.227605 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:55:28.227610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:55:28.227782 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:28.228501 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:55:28.228524 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:55:28.230741 (+ 2217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:55:28.235769 (+ 5028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:55:28.235782 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:55:28.235801 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:55:28.236478 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:55:28.236950 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:55:28.236958 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:55:28.236963 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:55:28.237221 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:55:28.238079 (+ 858us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:55:28.238091 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:55:28.238532 (+ 441us) client_negotiation.cc:769] Sending connection context
0929 15:55:28.238874 (+ 342us) client_negotiation.cc:240] Negotiation successful
0929 15:55:28.239517 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2873,"thread_start_us":2638,"threads_started":1}
I20260929 15:55:28.244053 16625 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38491
I20260929 15:55:28.244402 16625 heartbeater.cc:464] Registering TS with master...
I20260929 15:55:28.245167 16625 heartbeater.cc:511] Master 127.0.97.62:38491 requested a full tablet report, sending...
I20260929 15:55:28.247584 16440 ts_manager.cc:194] Registered new tserver with Master: 725fc31fc8b241cb8990f3e8cfb8ac44 (127.0.97.1:35363)
I20260929 15:55:28.249747 16440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34677
W20260929 15:55:28.543290 16629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:55:28.543609 16629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:55:28.543704 16629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:55:28.550803 16629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:55:28.550908 16629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:55:28.551016 16629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:55:28.560266 16629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:37361
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=33541
--tserver_master_addrs=127.0.97.62:38491
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:55:28.561406 16629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:55:28.562598 16629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:55:28.575078 16634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:55:28.575127 16635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:55:28.575151 16637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:55:28.576110 16629 server_base.cc:1034] running on GCE node
I20260929 15:55:28.576682 16629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:55:28.577661 16629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:55:28.578851 16629 hybrid_clock.cc:648] HybridClock initialized: now 1790697328578815 us; error 33 us; skew 500 ppm
I20260929 15:55:28.579154 16629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:55:28 dist-test-slave-r0ck krb5kdc[15856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:55:28.582167 16629 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:55:28.583469 16629 webserver.cc:460] Webserver started at http://127.0.97.2:33541/ using document root <none> and password file <none>
I20260929 15:55:28.584105 16629 fs_manager.cc:359] Metadata directory not provided
I20260929 15:55:28.584249 16629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:55:28.589284 16629 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260929 15:55:28.591770 16644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:55:28.593021 16629 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 15:55:28.593191 16629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/wal
uuid: "b2cce6a0fc254095826871e8d5a0a0fc"
format_stamp: "Formatted at 2026-09-29 15:55:05 on dist-test-slave-r0ck"
server_key: "724f69044ce0f997bb65f206403f49ba"
server_key_iv: "877f61eb7d1787f7c87f742607d0b2ac"
server_key_version: "encryptionkey@0"
I20260929 15:55:28.594156 16629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:55:28.619019 16629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:55:28.623234 16629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:55:28.623507 16629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:55:28.624437 16629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:55:28.625792 16629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:55:28.625910 16629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:55:28.626039 16629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:55:28.626127 16629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:55:28.659066 16629 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37361
I20260929 15:55:28.659164 16756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37361 every 8 connection(s)
I20260929 15:55:28.660418 16629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:55:28.668889 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 16629
I20260929 15:55:28.669656 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:39479
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:38491
--webserver_port=34951
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 15:55:28 dist-test-slave-r0ck krb5kdc[15856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:55:28.681785 16627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:28.662845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54991 (local address 127.0.97.62:38491)
0929 15:55:28.663067 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 15:55:28.663073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:55:28.664146 (+ 1073us) server_negotiation.cc:407] Connection header received
0929 15:55:28.665967 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:55:28.665972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:55:28.666037 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:55:28.666124 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:55:28.667493 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:28.668172 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:28.670589 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:28.670999 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:28.671092 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:28.675402 (+ 4310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:55:28.675408 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:55:28.675411 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:55:28.675444 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:55:28.678577 (+ 3133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:55:28.679296 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:55:28.679304 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:55:28.679310 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:55:28.679455 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:55:28.679845 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:55:28.679852 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:55:28.679859 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:55:28.680241 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:55:28.680387 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 15:55:28.681049 (+ 662us) server_negotiation.cc:299] Negotiation successful
0929 15:55:28.681306 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 15:55:28.682394 16759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:28.663284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.2:54991)
0929 15:55:28.663989 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:28.664034 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 15:55:28.665728 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:55:28.666309 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:55:28.666345 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:55:28.666819 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:55:28.667337 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:55:28.667353 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:55:28.668343 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:55:28.668368 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:55:28.670442 (+ 2074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:55:28.670460 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:55:28.671218 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:55:28.671228 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:55:28.671544 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:28.672265 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:55:28.672297 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:55:28.675161 (+ 2864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:55:28.678711 (+ 3550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:55:28.678718 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:55:28.678728 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:55:28.679165 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:55:28.679564 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:55:28.679568 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:55:28.679570 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:55:28.679732 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:55:28.680361 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:55:28.680366 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:55:28.680646 (+ 280us) client_negotiation.cc:769] Sending connection context
0929 15:55:28.680896 (+ 250us) client_negotiation.cc:240] Negotiation successful
0929 15:55:28.681253 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":292,"threads_started":1}
I20260929 15:55:28.683785 16757 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38491
I20260929 15:55:28.684087 16757 heartbeater.cc:464] Registering TS with master...
I20260929 15:55:28.684743 16757 heartbeater.cc:511] Master 127.0.97.62:38491 requested a full tablet report, sending...
I20260929 15:55:28.686154 16440 ts_manager.cc:194] Registered new tserver with Master: b2cce6a0fc254095826871e8d5a0a0fc (127.0.97.2:37361)
I20260929 15:55:28.686944 16440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:54991
W20260929 15:55:28.998126 16760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:55:28.998447 16760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:55:28.998607 16760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:55:29.006104 16760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:55:29.006258 16760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:55:29.006467 16760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:55:29.016544 16760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:39479
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=34951
--tserver_master_addrs=127.0.97.62:38491
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:55:29.017952 16760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:55:29.019233 16760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:55:29.032083 16766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:55:29.032130 16768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:55:29.032177 16765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:55:29.033018 16760 server_base.cc:1034] running on GCE node
I20260929 15:55:29.033654 16760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:55:29.034514 16760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:55:29.035748 16760 hybrid_clock.cc:648] HybridClock initialized: now 1790697329035710 us; error 66 us; skew 500 ppm
I20260929 15:55:29.036046 16760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:55:29 dist-test-slave-r0ck krb5kdc[15856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:55:29.039014 16760 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:55:29.040470 16760 webserver.cc:460] Webserver started at http://127.0.97.3:34951/ using document root <none> and password file <none>
I20260929 15:55:29.041122 16760 fs_manager.cc:359] Metadata directory not provided
I20260929 15:55:29.041280 16760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:55:29.046517 16760 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 15:55:29.049063 16775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:55:29.050439 16760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 15:55:29.050616 16760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/wal
uuid: "129d2804e3d54d01ad6eef1c52714dc9"
format_stamp: "Formatted at 2026-09-29 15:55:06 on dist-test-slave-r0ck"
server_key: "2305eba166af36f9aa4e4e048b2da4f6"
server_key_iv: "6546181ab497dccfbdc51021702bded9"
server_key_version: "encryptionkey@0"
I20260929 15:55:29.051592 16760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:55:29.073220 16760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:55:29.077123 16760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:55:29.077466 16760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:55:29.078449 16760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:55:29.079742 16760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:55:29.079865 16760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:55:29.080004 16760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:55:29.080093 16760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:55:29.114059 16760 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39479
I20260929 15:55:29.114188 16887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39479 every 8 connection(s)
I20260929 15:55:29.115579 16760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:55:29.125074 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 16760
Sep 29 15:55:29 dist-test-slave-r0ck krb5kdc[15856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:55:29.135876 16627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:29.118123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41615 (local address 127.0.97.62:38491)
0929 15:55:29.118307 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 15:55:29.118311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:55:29.119330 (+ 1019us) server_negotiation.cc:407] Connection header received
0929 15:55:29.121041 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:55:29.121046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:55:29.121111 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:55:29.121199 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:55:29.122663 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:29.123393 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:29.124843 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:29.125335 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:29.125523 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:29.129610 (+ 4087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:55:29.129614 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:55:29.129617 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:55:29.129647 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:55:29.132868 (+ 3221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:55:29.133716 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:55:29.133724 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:55:29.133728 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:55:29.133880 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:55:29.134275 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:55:29.134282 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:55:29.134286 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:55:29.134643 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:55:29.134806 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 15:55:29.135383 (+ 577us) server_negotiation.cc:299] Negotiation successful
0929 15:55:29.135591 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 15:55:29.137027 16890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:29.118496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.3:41615)
0929 15:55:29.119169 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:29.119204 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 15:55:29.120788 (+ 1584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:55:29.121457 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:55:29.121465 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:55:29.121988 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:55:29.122503 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:55:29.122519 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:55:29.123525 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:55:29.123529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:55:29.124678 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:55:29.124689 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:55:29.125616 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:55:29.125624 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:55:29.125864 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:29.126639 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:55:29.126675 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:55:29.129427 (+ 2752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:55:29.133025 (+ 3598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:55:29.133036 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:55:29.133051 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:55:29.133601 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:55:29.133991 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:55:29.133994 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:55:29.133997 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:55:29.134152 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:55:29.134766 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:55:29.134772 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:55:29.135041 (+ 269us) client_negotiation.cc:769] Sending connection context
0929 15:55:29.135243 (+ 202us) client_negotiation.cc:240] Negotiation successful
0929 15:55:29.135704 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":264,"threads_started":1}
I20260929 15:55:29.138710 16888 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38491
I20260929 15:55:29.138988 16888 heartbeater.cc:464] Registering TS with master...
I20260929 15:55:29.139639 16888 heartbeater.cc:511] Master 127.0.97.62:38491 requested a full tablet report, sending...
I20260929 15:55:29.141165 16440 ts_manager.cc:194] Registered new tserver with Master: 129d2804e3d54d01ad6eef1c52714dc9 (127.0.97.3:39479)
I20260929 15:55:29.141584 16892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:29.126054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41198 (local address 127.0.97.62:38491)
0929 15:55:29.126620 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0929 15:55:29.126625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:55:29.126643 (+ 18us) server_negotiation.cc:407] Connection header received
0929 15:55:29.126795 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:55:29.126799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:55:29.126881 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:55:29.127000 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:55:29.128794 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:29.129851 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:29.131955 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:29.132411 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:29.132504 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:29.135576 (+ 3072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:55:29.135584 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:55:29.135588 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:55:29.135627 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:55:29.138877 (+ 3250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:55:29.139551 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:55:29.139559 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:55:29.139562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:55:29.139683 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:55:29.140063 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:55:29.140068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:55:29.140071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:55:29.140374 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:55:29.140647 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0929 15:55:29.140950 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 15:55:29.141146 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":272,"threads_started":1}
I20260929 15:55:29.142248 16440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:41615
I20260929 15:55:29.144143 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:55:29.150231 16892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:29.145075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41214 (local address 127.0.97.62:38491)
0929 15:55:29.145219 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 15:55:29.145222 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 15:55:29.145711 (+ 489us) server_negotiation.cc:407] Connection header received
0929 15:55:29.145872 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:55:29.145875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:55:29.146008 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:55:29.146117 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:55:29.146480 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:29.147143 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:29.148507 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:29.148900 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:29.148988 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:29.149603 (+ 615us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:55:29.149776 (+ 173us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 15:55:29.150051 (+ 275us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41214: token expired
Metrics: {"server-negotiator.queue_time_us":53}
W20260929 15:55:29.150468 16892 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41214: token expired
W20260929 15:55:29.150960 16362 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:38491)
I20260929 15:55:29.163157 16892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:29.151460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41222 (local address 127.0.97.62:38491)
0929 15:55:29.151601 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 15:55:29.151605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:55:29.151619 (+ 14us) server_negotiation.cc:407] Connection header received
0929 15:55:29.151732 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:55:29.151735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:55:29.151784 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:55:29.151856 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:55:29.153200 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:29.153932 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:29.155309 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:29.155673 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:29.155763 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:29.158373 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:55:29.158377 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:55:29.158381 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:55:29.158411 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:55:29.160867 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:55:29.161534 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:55:29.161538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:55:29.161540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:55:29.161657 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:55:29.162034 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:55:29.162041 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:55:29.162047 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:55:29.162329 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:55:29.162514 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 15:55:29.162761 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 15:55:29.162922 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 15:55:29.166589 16440 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41222:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:55:29.173700 16440 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:55:29.196980 16901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:29.185807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38204 (local address 127.0.97.1:35363)
0929 15:55:29.188744 (+ 2937us) server_negotiation.cc:206] Beginning negotiation
0929 15:55:29.188754 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:55:29.188782 (+ 28us) server_negotiation.cc:407] Connection header received
0929 15:55:29.188867 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:55:29.188874 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:55:29.189081 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:55:29.189270 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:55:29.189934 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:29.191538 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:29.193949 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:29.195627 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:29.195789 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:29.196010 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 15:55:29.196447 (+ 437us) server_negotiation.cc:299] Negotiation successful
0929 15:55:29.196623 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2744,"thread_start_us":2636,"threads_started":1}
I20260929 15:55:29.197085 16900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:29.185109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35363 (local address 127.0.0.1:38204)
0929 15:55:29.186807 (+ 1698us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:29.186835 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 15:55:29.186962 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:55:29.189283 (+ 2321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:55:29.189288 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:55:29.189358 (+ 70us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:55:29.189778 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:55:29.189792 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:55:29.191691 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:55:29.191697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:55:29.193786 (+ 2089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:55:29.193804 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:55:29.195898 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:55:29.195905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:55:29.196151 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:29.196218 (+ 67us) client_negotiation.cc:769] Sending connection context
0929 15:55:29.196368 (+ 150us) client_negotiation.cc:240] Negotiation successful
0929 15:55:29.196567 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1546,"mutex_wait_us":11,"spinlock_wait_cycles":896,"thread_start_us":1778,"threads_started":1}
I20260929 15:55:29.197693 16898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:29.184729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39479 (local address 127.0.0.1:56776)
0929 15:55:29.185829 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:29.185863 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 15:55:29.186026 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:55:29.189807 (+ 3781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:55:29.189827 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:55:29.189853 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:55:29.190154 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:55:29.190161 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:55:29.192740 (+ 2579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:55:29.192745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:55:29.194975 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:55:29.194989 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:55:29.196867 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:55:29.196873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:55:29.197082 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:29.197123 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 15:55:29.197197 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 15:55:29.197324 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":887,"thread_start_us":769,"threads_started":1}
I20260929 15:55:29.197731 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:29.186533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40026 (local address 127.0.97.2:37361)
0929 15:55:29.187556 (+ 1023us) server_negotiation.cc:206] Beginning negotiation
0929 15:55:29.187568 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:55:29.188085 (+ 517us) server_negotiation.cc:407] Connection header received
0929 15:55:29.188394 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:55:29.188405 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:55:29.188654 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:55:29.188834 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:55:29.189922 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:29.191538 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:29.194406 (+ 2868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:29.195853 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:29.195954 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:29.196136 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 15:55:29.197073 (+ 937us) server_negotiation.cc:299] Negotiation successful
0929 15:55:29.197319 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":810,"thread_start_us":656,"threads_started":1}
I20260929 15:55:29.198038 16902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:29.186203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37361 (local address 127.0.0.1:40026)
0929 15:55:29.187988 (+ 1785us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:29.188004 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:55:29.188180 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:55:29.189119 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:55:29.189128 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:55:29.189156 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:55:29.189655 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:55:29.189689 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:55:29.191729 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:55:29.191733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:55:29.194220 (+ 2487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:55:29.194234 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:55:29.196274 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:55:29.196280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:55:29.196440 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:29.196469 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 15:55:29.196544 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 15:55:29.196642 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1677,"mutex_wait_us":100,"spinlock_wait_cycles":22272,"thread_start_us":1548,"threads_started":1}
I20260929 15:55:29.198783 16899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:29.184955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56776 (local address 127.0.97.3:39479)
0929 15:55:29.189183 (+ 4228us) server_negotiation.cc:206] Beginning negotiation
0929 15:55:29.189189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:55:29.189242 (+ 53us) server_negotiation.cc:407] Connection header received
0929 15:55:29.189333 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:55:29.189340 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:55:29.189582 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:55:29.190305 (+ 723us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:55:29.190572 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:29.192583 (+ 2011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:29.195163 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:29.196622 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:29.198037 (+ 1415us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:29.198249 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 15:55:29.198315 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 15:55:29.198466 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4090,"thread_start_us":3951,"threads_started":1}
I20260929 15:55:29.201553 16692 tablet_service.cc:1467] Processing CreateTablet for tablet 40c5d15029834286b3995273ac3856db (DEFAULT_TABLE table=test-table [id=0dae2793107841fda6aedf580b4cab8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:55:29.201551 16560 tablet_service.cc:1467] Processing CreateTablet for tablet 40c5d15029834286b3995273ac3856db (DEFAULT_TABLE table=test-table [id=0dae2793107841fda6aedf580b4cab8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:55:29.201882 16823 tablet_service.cc:1467] Processing CreateTablet for tablet 40c5d15029834286b3995273ac3856db (DEFAULT_TABLE table=test-table [id=0dae2793107841fda6aedf580b4cab8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:55:29.203009 16560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40c5d15029834286b3995273ac3856db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:55:29.203008 16692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40c5d15029834286b3995273ac3856db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:55:29.203188 16823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40c5d15029834286b3995273ac3856db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:55:29.221362 16905 tablet_bootstrap.cc:492] T 40c5d15029834286b3995273ac3856db P b2cce6a0fc254095826871e8d5a0a0fc: Bootstrap starting.
I20260929 15:55:29.224361 16906 tablet_bootstrap.cc:492] T 40c5d15029834286b3995273ac3856db P 725fc31fc8b241cb8990f3e8cfb8ac44: Bootstrap starting.
I20260929 15:55:29.227571 16904 tablet_bootstrap.cc:492] T 40c5d15029834286b3995273ac3856db P 129d2804e3d54d01ad6eef1c52714dc9: Bootstrap starting.
I20260929 15:55:29.230000 16906 tablet_bootstrap.cc:654] T 40c5d15029834286b3995273ac3856db P 725fc31fc8b241cb8990f3e8cfb8ac44: Neither blocks nor log segments found. Creating new log.
I20260929 15:55:29.230647 16905 tablet_bootstrap.cc:654] T 40c5d15029834286b3995273ac3856db P b2cce6a0fc254095826871e8d5a0a0fc: Neither blocks nor log segments found. Creating new log.
I20260929 15:55:29.231277 16906 log.cc:824] T 40c5d15029834286b3995273ac3856db P 725fc31fc8b241cb8990f3e8cfb8ac44: Log is configured to *not* fsync() on all Append() calls
I20260929 15:55:29.231742 16905 log.cc:824] T 40c5d15029834286b3995273ac3856db P b2cce6a0fc254095826871e8d5a0a0fc: Log is configured to *not* fsync() on all Append() calls
I20260929 15:55:29.232717 16904 tablet_bootstrap.cc:654] T 40c5d15029834286b3995273ac3856db P 129d2804e3d54d01ad6eef1c52714dc9: Neither blocks nor log segments found. Creating new log.
I20260929 15:55:29.233704 16904 log.cc:824] T 40c5d15029834286b3995273ac3856db P 129d2804e3d54d01ad6eef1c52714dc9: Log is configured to *not* fsync() on all Append() calls
I20260929 15:55:29.234165 16906 tablet_bootstrap.cc:492] T 40c5d15029834286b3995273ac3856db P 725fc31fc8b241cb8990f3e8cfb8ac44: No bootstrap required, opened a new log
I20260929 15:55:29.234483 16906 ts_tablet_manager.cc:1397] T 40c5d15029834286b3995273ac3856db P 725fc31fc8b241cb8990f3e8cfb8ac44: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260929 15:55:29.236142 16905 tablet_bootstrap.cc:492] T 40c5d15029834286b3995273ac3856db P b2cce6a0fc254095826871e8d5a0a0fc: No bootstrap required, opened a new log
I20260929 15:55:29.236589 16904 tablet_bootstrap.cc:492] T 40c5d15029834286b3995273ac3856db P 129d2804e3d54d01ad6eef1c52714dc9: No bootstrap required, opened a new log
I20260929 15:55:29.236842 16904 ts_tablet_manager.cc:1397] T 40c5d15029834286b3995273ac3856db P 129d2804e3d54d01ad6eef1c52714dc9: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 15:55:29.237000 16905 ts_tablet_manager.cc:1397] T 40c5d15029834286b3995273ac3856db P b2cce6a0fc254095826871e8d5a0a0fc: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.000s
I20260929 15:55:29.245718 16905 raft_consensus.cc:348] T 40c5d15029834286b3995273ac3856db P b2cce6a0fc254095826871e8d5a0a0fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129d2804e3d54d01ad6eef1c52714dc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39479 } } peers { permanent_uuid: "725fc31fc8b241cb8990f3e8cfb8ac44" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35363 } } peers { permanent_uuid: "b2cce6a0fc254095826871e8d5a0a0fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37361 } }
I20260929 15:55:29.246150 16905 raft_consensus.cc:374] T 40c5d15029834286b3995273ac3856db P b2cce6a0fc254095826871e8d5a0a0fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:55:29.245929 16906 raft_consensus.cc:348] T 40c5d15029834286b3995273ac3856db P 725fc31fc8b241cb8990f3e8cfb8ac44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129d2804e3d54d01ad6eef1c52714dc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39479 } } peers { permanent_uuid: "725fc31fc8b241cb8990f3e8cfb8ac44" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35363 } } peers { permanent_uuid: "b2cce6a0fc254095826871e8d5a0a0fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37361 } }
I20260929 15:55:29.246243 16905 raft_consensus.cc:729] T 40c5d15029834286b3995273ac3856db P b2cce6a0fc254095826871e8d5a0a0fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2cce6a0fc254095826871e8d5a0a0fc, State: Initialized, Role: FOLLOWER
I20260929 15:55:29.246364 16906 raft_consensus.cc:374] T 40c5d15029834286b3995273ac3856db P 725fc31fc8b241cb8990f3e8cfb8ac44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:55:29.246470 16906 raft_consensus.cc:729] T 40c5d15029834286b3995273ac3856db P 725fc31fc8b241cb8990f3e8cfb8ac44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 725fc31fc8b241cb8990f3e8cfb8ac44, State: Initialized, Role: FOLLOWER
I20260929 15:55:29.246899 16905 consensus_queue.cc:260] T 40c5d15029834286b3995273ac3856db P b2cce6a0fc254095826871e8d5a0a0fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129d2804e3d54d01ad6eef1c52714dc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39479 } } peers { permanent_uuid: "725fc31fc8b241cb8990f3e8cfb8ac44" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35363 } } peers { permanent_uuid: "b2cce6a0fc254095826871e8d5a0a0fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37361 } }
I20260929 15:55:29.247112 16906 consensus_queue.cc:260] T 40c5d15029834286b3995273ac3856db P 725fc31fc8b241cb8990f3e8cfb8ac44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129d2804e3d54d01ad6eef1c52714dc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39479 } } peers { permanent_uuid: "725fc31fc8b241cb8990f3e8cfb8ac44" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35363 } } peers { permanent_uuid: "b2cce6a0fc254095826871e8d5a0a0fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37361 } }
I20260929 15:55:29.247582 16904 raft_consensus.cc:348] T 40c5d15029834286b3995273ac3856db P 129d2804e3d54d01ad6eef1c52714dc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129d2804e3d54d01ad6eef1c52714dc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39479 } } peers { permanent_uuid: "725fc31fc8b241cb8990f3e8cfb8ac44" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35363 } } peers { permanent_uuid: "b2cce6a0fc254095826871e8d5a0a0fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37361 } }
I20260929 15:55:29.248111 16904 raft_consensus.cc:374] T 40c5d15029834286b3995273ac3856db P 129d2804e3d54d01ad6eef1c52714dc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:55:29.248126 16757 heartbeater.cc:503] Master 127.0.97.62:38491 was elected leader, sending a full tablet report...
I20260929 15:55:29.248240 16904 raft_consensus.cc:729] T 40c5d15029834286b3995273ac3856db P 129d2804e3d54d01ad6eef1c52714dc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129d2804e3d54d01ad6eef1c52714dc9, State: Initialized, Role: FOLLOWER
I20260929 15:55:29.248947 16904 consensus_queue.cc:260] T 40c5d15029834286b3995273ac3856db P 129d2804e3d54d01ad6eef1c52714dc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129d2804e3d54d01ad6eef1c52714dc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39479 } } peers { permanent_uuid: "725fc31fc8b241cb8990f3e8cfb8ac44" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35363 } } peers { permanent_uuid: "b2cce6a0fc254095826871e8d5a0a0fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37361 } }
I20260929 15:55:29.250231 16905 ts_tablet_manager.cc:1428] T 40c5d15029834286b3995273ac3856db P b2cce6a0fc254095826871e8d5a0a0fc: Time spent starting tablet: real 0.013s user 0.007s sys 0.004s
I20260929 15:55:29.251120 16625 heartbeater.cc:503] Master 127.0.97.62:38491 was elected leader, sending a full tablet report...
I20260929 15:55:29.251953 16906 ts_tablet_manager.cc:1428] T 40c5d15029834286b3995273ac3856db P 725fc31fc8b241cb8990f3e8cfb8ac44: Time spent starting tablet: real 0.017s user 0.013s sys 0.003s
I20260929 15:55:29.253003 16888 heartbeater.cc:503] Master 127.0.97.62:38491 was elected leader, sending a full tablet report...
I20260929 15:55:29.253674 16904 ts_tablet_manager.cc:1428] T 40c5d15029834286b3995273ac3856db P 129d2804e3d54d01ad6eef1c52714dc9: Time spent starting tablet: real 0.017s user 0.012s sys 0.005s
I20260929 15:55:29.335501 16912 raft_consensus.cc:482] T 40c5d15029834286b3995273ac3856db P 129d2804e3d54d01ad6eef1c52714dc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:55:29.335842 16912 raft_consensus.cc:504] T 40c5d15029834286b3995273ac3856db P 129d2804e3d54d01ad6eef1c52714dc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129d2804e3d54d01ad6eef1c52714dc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39479 } } peers { permanent_uuid: "725fc31fc8b241cb8990f3e8cfb8ac44" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35363 } } peers { permanent_uuid: "b2cce6a0fc254095826871e8d5a0a0fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37361 } }
I20260929 15:55:29.337612 16912 leader_election.cc:288] T 40c5d15029834286b3995273ac3856db P 129d2804e3d54d01ad6eef1c52714dc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 725fc31fc8b241cb8990f3e8cfb8ac44 (127.0.97.1:35363), b2cce6a0fc254095826871e8d5a0a0fc (127.0.97.2:37361)
I20260929 15:55:29.343509 16901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:29.338073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37017 (local address 127.0.97.1:35363)
0929 15:55:29.338314 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 15:55:29.338322 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:55:29.338431 (+ 109us) server_negotiation.cc:407] Connection header received
0929 15:55:29.338667 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:55:29.338673 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:55:29.338733 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:55:29.338835 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:55:29.339235 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:29.339939 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:29.341708 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:29.342566 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:29.342708 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:29.342801 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 15:55:29.343110 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 15:55:29.343232 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260929 15:55:29.343547 16890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:29.337983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35363 (local address 127.0.97.3:37017)
0929 15:55:29.338312 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:29.338332 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:55:29.338512 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:55:29.338861 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:55:29.338865 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:55:29.338901 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:55:29.339095 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:55:29.339103 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:55:29.340094 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:55:29.340099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:55:29.341557 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:55:29.341569 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:55:29.342717 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:55:29.342723 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:55:29.342936 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:29.342975 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 15:55:29.343074 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 15:55:29.343205 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":35}
I20260929 15:55:29.344786 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:29.338111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42485 (local address 127.0.97.2:37361)
0929 15:55:29.338319 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 15:55:29.338326 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:55:29.339079 (+ 753us) server_negotiation.cc:407] Connection header received
0929 15:55:29.339277 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:55:29.339289 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:55:29.339353 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:55:29.339451 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:55:29.339996 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:29.340691 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:29.342541 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:29.343378 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:29.343533 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:29.343657 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 15:55:29.344395 (+ 738us) server_negotiation.cc:299] Negotiation successful
0929 15:55:29.344527 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 15:55:29.344719 16580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c5d15029834286b3995273ac3856db" candidate_uuid: "129d2804e3d54d01ad6eef1c52714dc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "725fc31fc8b241cb8990f3e8cfb8ac44" is_pre_election: true
I20260929 15:55:29.345018 16913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:29.338144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37361 (local address 127.0.97.3:42485)
0929 15:55:29.338963 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:29.338978 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 15:55:29.339133 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:55:29.339462 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:55:29.339467 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:55:29.339488 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:55:29.339833 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:55:29.339841 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:55:29.340858 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:55:29.340868 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:55:29.342385 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:55:29.342397 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:55:29.344001 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:55:29.344011 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:55:29.344208 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:29.344242 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 15:55:29.344354 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 15:55:29.344472 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"mutex_wait_us":153,"thread_start_us":460,"threads_started":1}
I20260929 15:55:29.345395 16580 raft_consensus.cc:2454] T 40c5d15029834286b3995273ac3856db P 725fc31fc8b241cb8990f3e8cfb8ac44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 129d2804e3d54d01ad6eef1c52714dc9 in term 0.
I20260929 15:55:29.345924 16712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c5d15029834286b3995273ac3856db" candidate_uuid: "129d2804e3d54d01ad6eef1c52714dc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2cce6a0fc254095826871e8d5a0a0fc" is_pre_election: true
I20260929 15:55:29.346123 16779 leader_election.cc:302] T 40c5d15029834286b3995273ac3856db P 129d2804e3d54d01ad6eef1c52714dc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 129d2804e3d54d01ad6eef1c52714dc9, 725fc31fc8b241cb8990f3e8cfb8ac44; no voters:
I20260929 15:55:29.346390 16712 raft_consensus.cc:2454] T 40c5d15029834286b3995273ac3856db P b2cce6a0fc254095826871e8d5a0a0fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 129d2804e3d54d01ad6eef1c52714dc9 in term 0.
I20260929 15:55:29.346544 16912 raft_consensus.cc:2781] T 40c5d15029834286b3995273ac3856db P 129d2804e3d54d01ad6eef1c52714dc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:55:29.346692 16912 raft_consensus.cc:482] T 40c5d15029834286b3995273ac3856db P 129d2804e3d54d01ad6eef1c52714dc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:55:29.346812 16912 raft_consensus.cc:3037] T 40c5d15029834286b3995273ac3856db P 129d2804e3d54d01ad6eef1c52714dc9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:55:29.349027 16912 raft_consensus.cc:504] T 40c5d15029834286b3995273ac3856db P 129d2804e3d54d01ad6eef1c52714dc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129d2804e3d54d01ad6eef1c52714dc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39479 } } peers { permanent_uuid: "725fc31fc8b241cb8990f3e8cfb8ac44" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35363 } } peers { permanent_uuid: "b2cce6a0fc254095826871e8d5a0a0fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37361 } }
I20260929 15:55:29.349740 16912 leader_election.cc:288] T 40c5d15029834286b3995273ac3856db P 129d2804e3d54d01ad6eef1c52714dc9 [CANDIDATE]: Term 1 election: Requested vote from peers 725fc31fc8b241cb8990f3e8cfb8ac44 (127.0.97.1:35363), b2cce6a0fc254095826871e8d5a0a0fc (127.0.97.2:37361)
I20260929 15:55:29.350257 16712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c5d15029834286b3995273ac3856db" candidate_uuid: "129d2804e3d54d01ad6eef1c52714dc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2cce6a0fc254095826871e8d5a0a0fc"
I20260929 15:55:29.350452 16712 raft_consensus.cc:3037] T 40c5d15029834286b3995273ac3856db P b2cce6a0fc254095826871e8d5a0a0fc [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:55:29.350453 16580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c5d15029834286b3995273ac3856db" candidate_uuid: "129d2804e3d54d01ad6eef1c52714dc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "725fc31fc8b241cb8990f3e8cfb8ac44"
I20260929 15:55:29.350675 16580 raft_consensus.cc:3037] T 40c5d15029834286b3995273ac3856db P 725fc31fc8b241cb8990f3e8cfb8ac44 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:55:29.352648 16712 raft_consensus.cc:2454] T 40c5d15029834286b3995273ac3856db P b2cce6a0fc254095826871e8d5a0a0fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 129d2804e3d54d01ad6eef1c52714dc9 in term 1.
I20260929 15:55:29.352860 16580 raft_consensus.cc:2454] T 40c5d15029834286b3995273ac3856db P 725fc31fc8b241cb8990f3e8cfb8ac44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 129d2804e3d54d01ad6eef1c52714dc9 in term 1.
I20260929 15:55:29.353073 16777 leader_election.cc:302] T 40c5d15029834286b3995273ac3856db P 129d2804e3d54d01ad6eef1c52714dc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 129d2804e3d54d01ad6eef1c52714dc9, b2cce6a0fc254095826871e8d5a0a0fc; no voters:
I20260929 15:55:29.353417 16912 raft_consensus.cc:2781] T 40c5d15029834286b3995273ac3856db P 129d2804e3d54d01ad6eef1c52714dc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:55:29.353809 16912 raft_consensus.cc:686] T 40c5d15029834286b3995273ac3856db P 129d2804e3d54d01ad6eef1c52714dc9 [term 1 LEADER]: Becoming Leader. State: Replica: 129d2804e3d54d01ad6eef1c52714dc9, State: Running, Role: LEADER
I20260929 15:55:29.354257 16912 consensus_queue.cc:237] T 40c5d15029834286b3995273ac3856db P 129d2804e3d54d01ad6eef1c52714dc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129d2804e3d54d01ad6eef1c52714dc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39479 } } peers { permanent_uuid: "725fc31fc8b241cb8990f3e8cfb8ac44" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35363 } } peers { permanent_uuid: "b2cce6a0fc254095826871e8d5a0a0fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37361 } }
I20260929 15:55:29.358649 16437 catalog_manager.cc:5495] T 40c5d15029834286b3995273ac3856db P 129d2804e3d54d01ad6eef1c52714dc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 129d2804e3d54d01ad6eef1c52714dc9 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "129d2804e3d54d01ad6eef1c52714dc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129d2804e3d54d01ad6eef1c52714dc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "725fc31fc8b241cb8990f3e8cfb8ac44" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2cce6a0fc254095826871e8d5a0a0fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37361 } health_report { overall_health: UNKNOWN } } }
W20260929 15:55:29.367089 16889 tablet.cc:2369] T 40c5d15029834286b3995273ac3856db P 129d2804e3d54d01ad6eef1c52714dc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:55:29.393603 16899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:29.388154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56784 (local address 127.0.97.3:39479)
0929 15:55:29.388421 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0929 15:55:29.388428 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:55:29.388525 (+ 97us) server_negotiation.cc:407] Connection header received
0929 15:55:29.388776 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:55:29.388782 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:55:29.388865 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:55:29.389013 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:55:29.389415 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:29.390100 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:29.391513 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:29.391912 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:29.392046 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:29.392536 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:55:29.392999 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:55:29.393149 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 15:55:29.393229 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 15:55:29.393316 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 15:55:29.401904 16712 raft_consensus.cc:1261] T 40c5d15029834286b3995273ac3856db P b2cce6a0fc254095826871e8d5a0a0fc [term 1 FOLLOWER]: Refusing update from remote peer 129d2804e3d54d01ad6eef1c52714dc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:55:29.401904 16580 raft_consensus.cc:1261] T 40c5d15029834286b3995273ac3856db P 725fc31fc8b241cb8990f3e8cfb8ac44 [term 1 FOLLOWER]: Refusing update from remote peer 129d2804e3d54d01ad6eef1c52714dc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:55:29.403025 16914 consensus_queue.cc:1035] T 40c5d15029834286b3995273ac3856db P 129d2804e3d54d01ad6eef1c52714dc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "725fc31fc8b241cb8990f3e8cfb8ac44" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:55:29.403390 16912 consensus_queue.cc:1035] T 40c5d15029834286b3995273ac3856db P 129d2804e3d54d01ad6eef1c52714dc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2cce6a0fc254095826871e8d5a0a0fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 15:55:29.412308 16758 tablet.cc:2369] T 40c5d15029834286b3995273ac3856db P b2cce6a0fc254095826871e8d5a0a0fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:55:29.431748 16438 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:55:29.432093 16438 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:55:29.435962 16438 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9: Sending DeleteTablet for 3 replicas of tablet 40c5d15029834286b3995273ac3856db
I20260929 15:55:29.437152 16823 tablet_service.cc:1514] Processing DeleteTablet for tablet 40c5d15029834286b3995273ac3856db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:55:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56776
I20260929 15:55:29.437263 16560 tablet_service.cc:1514] Processing DeleteTablet for tablet 40c5d15029834286b3995273ac3856db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:55:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38204
I20260929 15:55:29.437412 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 16477
I20260929 15:55:29.437510 16692 tablet_service.cc:1514] Processing DeleteTablet for tablet 40c5d15029834286b3995273ac3856db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:55:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40026
I20260929 15:55:29.448103 16929 tablet_replica.cc:330] T 40c5d15029834286b3995273ac3856db P 725fc31fc8b241cb8990f3e8cfb8ac44: stopping tablet replica
I20260929 15:55:29.448670 16929 raft_consensus.cc:2229] T 40c5d15029834286b3995273ac3856db P 725fc31fc8b241cb8990f3e8cfb8ac44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:55:29.449301 16929 raft_consensus.cc:2258] T 40c5d15029834286b3995273ac3856db P 725fc31fc8b241cb8990f3e8cfb8ac44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:55:29.450309 16928 tablet_replica.cc:330] T 40c5d15029834286b3995273ac3856db P 129d2804e3d54d01ad6eef1c52714dc9: stopping tablet replica
I20260929 15:55:29.450711 16928 raft_consensus.cc:2229] T 40c5d15029834286b3995273ac3856db P 129d2804e3d54d01ad6eef1c52714dc9 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:55:29.451479 16928 raft_consensus.cc:2258] T 40c5d15029834286b3995273ac3856db P 129d2804e3d54d01ad6eef1c52714dc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:55:29.453922 16929 ts_tablet_manager.cc:1905] T 40c5d15029834286b3995273ac3856db P 725fc31fc8b241cb8990f3e8cfb8ac44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:55:29.457175 16928 ts_tablet_manager.cc:1905] T 40c5d15029834286b3995273ac3856db P 129d2804e3d54d01ad6eef1c52714dc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:55:29.457576 16901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:29.438649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38212 (local address 127.0.97.1:35363)
0929 15:55:29.438864 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 15:55:29.438873 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:55:29.438897 (+ 24us) server_negotiation.cc:407] Connection header received
0929 15:55:29.438967 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:55:29.438974 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:55:29.439109 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:55:29.439248 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:55:29.441590 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:29.443162 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:29.445313 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:29.446219 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:29.446383 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:29.450073 (+ 3690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:55:29.450086 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:55:29.450105 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:55:29.450146 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:55:29.453943 (+ 3797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:55:29.454766 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:55:29.454774 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:55:29.454777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:55:29.454914 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:55:29.455297 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:55:29.455315 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:55:29.455318 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:55:29.455877 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:55:29.456159 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0929 15:55:29.456632 (+ 473us) server_negotiation.cc:299] Negotiation successful
0929 15:55:29.456849 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 15:55:29.458190 16620 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:55:29.462393 16930 tablet_replica.cc:330] T 40c5d15029834286b3995273ac3856db P b2cce6a0fc254095826871e8d5a0a0fc: stopping tablet replica
I20260929 15:55:29.462913 16930 raft_consensus.cc:2229] T 40c5d15029834286b3995273ac3856db P b2cce6a0fc254095826871e8d5a0a0fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:55:29.463495 16930 raft_consensus.cc:2258] T 40c5d15029834286b3995273ac3856db P b2cce6a0fc254095826871e8d5a0a0fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:55:29.465267 16928 ts_tablet_manager.cc:1918] T 40c5d15029834286b3995273ac3856db P 129d2804e3d54d01ad6eef1c52714dc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:55:29.465538 16928 log.cc:1196] T 40c5d15029834286b3995273ac3856db P 129d2804e3d54d01ad6eef1c52714dc9: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/wal/wals/40c5d15029834286b3995273ac3856db
I20260929 15:55:29.465910 16930 ts_tablet_manager.cc:1905] T 40c5d15029834286b3995273ac3856db P b2cce6a0fc254095826871e8d5a0a0fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:55:29.466259 16928 ts_tablet_manager.cc:1939] T 40c5d15029834286b3995273ac3856db P 129d2804e3d54d01ad6eef1c52714dc9: Deleting consensus metadata
I20260929 15:55:29.468065 16425 catalog_manager.cc:4841] TS 129d2804e3d54d01ad6eef1c52714dc9 (127.0.97.3:39479): tablet 40c5d15029834286b3995273ac3856db (table test-table [id=0dae2793107841fda6aedf580b4cab8c]) successfully deleted
I20260929 15:55:29.471428 16930 ts_tablet_manager.cc:1918] T 40c5d15029834286b3995273ac3856db P b2cce6a0fc254095826871e8d5a0a0fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:55:29.471592 16930 log.cc:1196] T 40c5d15029834286b3995273ac3856db P b2cce6a0fc254095826871e8d5a0a0fc: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/wal/wals/40c5d15029834286b3995273ac3856db
I20260929 15:55:29.472060 16930 ts_tablet_manager.cc:1939] T 40c5d15029834286b3995273ac3856db P b2cce6a0fc254095826871e8d5a0a0fc: Deleting consensus metadata
I20260929 15:55:29.473601 16425 catalog_manager.cc:4841] TS b2cce6a0fc254095826871e8d5a0a0fc (127.0.97.2:37361): tablet 40c5d15029834286b3995273ac3856db (table test-table [id=0dae2793107841fda6aedf580b4cab8c]) successfully deleted
I20260929 15:55:29.581290 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 16477
I20260929 15:55:29.598932 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 16629
W20260929 15:55:29.599148 16427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:35363 (error 108)
I20260929 15:55:29.601393 16902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:29.600847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35363 (local address 127.0.0.1:38218)
0929 15:55:29.601041 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:29.601156 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260929 15:55:29.601724 16427 catalog_manager.cc:4584] TS 725fc31fc8b241cb8990f3e8cfb8ac44 (127.0.97.1:35363): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 40c5d15029834286b3995273ac3856db: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35363: connect: Connection refused (error 111)
I20260929 15:55:29.615238 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:29.599850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40030 (local address 127.0.97.2:37361)
0929 15:55:29.600033 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 15:55:29.600041 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:55:29.600067 (+ 26us) server_negotiation.cc:407] Connection header received
0929 15:55:29.600285 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:55:29.600292 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:55:29.600397 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:55:29.600502 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:55:29.602257 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:29.603420 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:29.604814 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:29.605564 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:29.605691 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:29.608488 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:55:29.608506 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:55:29.608525 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:55:29.608580 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:55:29.612169 (+ 3589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:55:29.612889 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:55:29.612900 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:55:29.612905 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:55:29.613060 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:55:29.613664 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:55:29.613670 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:55:29.613673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:55:29.614073 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:55:29.614285 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 15:55:29.614747 (+ 462us) server_negotiation.cc:299] Negotiation successful
0929 15:55:29.614914 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 15:55:29.615903 16752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:55:29.655483 16902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:29.654906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35363 (local address 127.0.0.1:38234)
0929 15:55:29.655173 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:29.655301 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260929 15:55:29.656700 16902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:29.656306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35363 (local address 127.0.0.1:38240)
0929 15:55:29.656468 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:29.656567 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 15:55:29.705590 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 16629
I20260929 15:55:29.717408 16902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:29.716824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35363 (local address 127.0.0.1:38242)
0929 15:55:29.717066 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:29.717181 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260929 15:55:29.718658 16902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:29.718218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35363 (local address 127.0.0.1:38246)
0929 15:55:29.718407 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:29.718501 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 15:55:29.723572 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 16760
I20260929 15:55:29.745877 16899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:29.728249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56786 (local address 127.0.97.3:39479)
0929 15:55:29.728431 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 15:55:29.728435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:55:29.728455 (+ 20us) server_negotiation.cc:407] Connection header received
0929 15:55:29.728516 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:55:29.728520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:55:29.728586 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:55:29.728672 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:55:29.730650 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:29.731970 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:29.733932 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:55:29.734612 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:55:29.734741 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:55:29.738412 (+ 3671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:55:29.738425 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:55:29.738446 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:55:29.738490 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:55:29.742446 (+ 3956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:55:29.743374 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:55:29.743386 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:55:29.743391 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:55:29.743607 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:55:29.744075 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:55:29.744080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:55:29.744084 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:55:29.744549 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:55:29.744845 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0929 15:55:29.745234 (+ 389us) server_negotiation.cc:299] Negotiation successful
0929 15:55:29.745486 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 15:55:29.746652 16883 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:55:29.799430 16902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:29.798889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35363 (local address 127.0.0.1:38260)
0929 15:55:29.799117 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:29.799225 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 15:55:29.800549 16902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:29.800109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35363 (local address 127.0.0.1:38262)
0929 15:55:29.800322 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:29.800418 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260929 15:55:29.873853 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 16760
I20260929 15:55:29.893500 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 16407
I20260929 15:55:29.894757 16470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:55:30.008889 16902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:30.008167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35363 (local address 127.0.0.1:38264)
0929 15:55:30.008471 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:30.008599 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260929 15:55:30.009027 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 16407
I20260929 15:55:50.032060 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38491
--webserver_interface=127.0.97.62
--webserver_port=34989
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:55:50.377053 16936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:55:50.377315 16936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:55:50.377512 16936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:55:50.385697 16936 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 15:55:50.385804 16936 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 15:55:50.385869 16936 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 15:55:50.385967 16936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:55:50.386072 16936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:55:50.386185 16936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:55:50.386255 16936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:55:50.386325 16936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:55:50.396423 16936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=34989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:55:50.397709 16936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:55:50.399123 16936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:55:50.411451 16942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:55:50.411571 16941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:55:50.411510 16944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:55:50.411767 16936 server_base.cc:1034] running on GCE node
I20260929 15:55:50.412892 16936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:55:50.414335 16936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:55:50.415550 16936 hybrid_clock.cc:648] HybridClock initialized: now 1790697350415492 us; error 30 us; skew 500 ppm
I20260929 15:55:50.415880 16936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:55:50 dist-test-slave-r0ck krb5kdc[15856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:55:50.419049 16936 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:55:50.420467 16936 webserver.cc:460] Webserver started at http://127.0.97.62:34989/ using document root <none> and password file <none>
I20260929 15:55:50.421113 16936 fs_manager.cc:359] Metadata directory not provided
I20260929 15:55:50.421197 16936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:55:50.426651 16936 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 15:55:50.429519 16951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:55:50.430845 16936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:55:50.431038 16936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "29da14e05d2b468ba3c91329fd95aff9"
format_stamp: "Formatted at 2026-09-29 15:55:04 on dist-test-slave-r0ck"
server_key: "f500f5b7e97e070a6e58e107e4e04045"
server_key_iv: "3bbd59075cf52066af916acb4c690389"
server_key_version: "encryptionkey@0"
I20260929 15:55:50.431975 16936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:55:50.450404 16936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:55:50.454517 16936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:55:50.454825 16936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:55:50.473136 16936 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38491
I20260929 15:55:50.473254 17002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38491 every 8 connection(s)
I20260929 15:55:50.474643 16936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:55:50.476851 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 16936
I20260929 15:55:50.477900 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:35363
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:38491
--webserver_port=36367
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:55:50.484373 17003 sys_catalog.cc:263] Verifying existing consensus state
I20260929 15:55:50.488607 17003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9: Bootstrap starting.
I20260929 15:55:50.508764 17003 log.cc:824] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9: Log is configured to *not* fsync() on all Append() calls
I20260929 15:55:50.524330 17003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 15:55:50.524999 17003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9: Bootstrap complete.
I20260929 15:55:50.533977 17003 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29da14e05d2b468ba3c91329fd95aff9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38491 } }
I20260929 15:55:50.534842 17003 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29da14e05d2b468ba3c91329fd95aff9, State: Initialized, Role: FOLLOWER
I20260929 15:55:50.535519 17003 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29da14e05d2b468ba3c91329fd95aff9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38491 } }
I20260929 15:55:50.535770 17003 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:55:50.535885 17003 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:55:50.536072 17003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [term 2 FOLLOWER]: Advancing to term 3
I20260929 15:55:50.539896 17003 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29da14e05d2b468ba3c91329fd95aff9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38491 } }
I20260929 15:55:50.540372 17003 leader_election.cc:302] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29da14e05d2b468ba3c91329fd95aff9; no voters:
I20260929 15:55:50.541044 17003 leader_election.cc:288] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 15:55:50.541188 17007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 15:55:50.542425 17007 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [term 3 LEADER]: Becoming Leader. State: Replica: 29da14e05d2b468ba3c91329fd95aff9, State: Running, Role: LEADER
I20260929 15:55:50.542961 17007 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29da14e05d2b468ba3c91329fd95aff9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38491 } }
I20260929 15:55:50.543203 17003 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:55:50.545760 17008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "29da14e05d2b468ba3c91329fd95aff9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29da14e05d2b468ba3c91329fd95aff9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38491 } } }
I20260929 15:55:50.546077 17008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [sys.catalog]: This master's current role is: LEADER
I20260929 15:55:50.545325 17009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29da14e05d2b468ba3c91329fd95aff9. Latest consensus state: current_term: 3 leader_uuid: "29da14e05d2b468ba3c91329fd95aff9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29da14e05d2b468ba3c91329fd95aff9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38491 } } }
I20260929 15:55:50.546625 17009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9 [sys.catalog]: This master's current role is: LEADER
I20260929 15:55:50.558418 17013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:55:50.568871 17013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:55:50.570760 17013 catalog_manager.cc:1259] Loaded cluster ID: b75c332d7dba4033b10c80d75da9bcb8
I20260929 15:55:50.570883 17013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:55:50.576140 17013 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:55:50.578176 17013 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9: Loaded TSK: 0
I20260929 15:55:50.578316 17013 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 29da14e05d2b468ba3c91329fd95aff9: Loaded TSK: 1
F20260929 15:55:50.579864 17013 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fcfb28099ad google::LogMessage::Fail() at ??:0
@ 0x7fcfb280c88a google::LogMessage::SendToLog() at ??:0
@ 0x7fcfb28094de google::LogMessage::Flush() at ??:0
@ 0x7fcfb280aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fcfb656d7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fcfb655c73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fcfc82381c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fcfc8237783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fcfc82c32a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
I20260929 15:55:50.618463 17026 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7fcfc82c30c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fcfc830d5e9 std::function<>::operator()() at ??:0
@ 0x7fcfc823bcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fcfc8239c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fcfc82c1610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fcfc82c1439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fcfb5455345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fcfb545891c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fcfb5458799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fcfb5433eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fcfb870d6db start_thread at ??:0
@ 0x7fcfb16a361f clone at ??:0
W20260929 15:55:50.892869 17005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:55:50.893154 17005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:55:50.893302 17005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:55:50.900652 17005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:55:50.900769 17005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:55:50.900934 17005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:55:50.910946 17005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:35363
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=36367
--tserver_master_addrs=127.0.97.62:38491
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:55:50.912909 17005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:55:50.914311 17005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:55:50.927362 17050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:55:50.927316 17053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:55:50.927349 17051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:55:50.928689 17005 server_base.cc:1034] running on GCE node
I20260929 15:55:50.929297 17005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:55:50.930287 17005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:55:50.931514 17005 hybrid_clock.cc:648] HybridClock initialized: now 1790697350931447 us; error 39 us; skew 500 ppm
I20260929 15:55:50.931833 17005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:55:50 dist-test-slave-r0ck krb5kdc[15856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:55:50.934845 17005 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:55:50.936287 17005 webserver.cc:460] Webserver started at http://127.0.97.1:36367/ using document root <none> and password file <none>
I20260929 15:55:50.936932 17005 fs_manager.cc:359] Metadata directory not provided
I20260929 15:55:50.937067 17005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:55:50.942202 17005 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:55:50.944911 17060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:55:50.946293 17005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 15:55:50.946457 17005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/wal
uuid: "725fc31fc8b241cb8990f3e8cfb8ac44"
format_stamp: "Formatted at 2026-09-29 15:55:05 on dist-test-slave-r0ck"
server_key: "51807be11621fed588914153c42c4e30"
server_key_iv: "9bb144e888d933f0b4c26c8ce5790384"
server_key_version: "encryptionkey@0"
I20260929 15:55:50.947507 17005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:55:50.962280 17005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:55:50.966325 17005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:55:50.966629 17005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:55:50.967631 17005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:55:50.969524 17067 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260929 15:55:50.974792 17067 ts_tablet_manager.cc:1852] T 40c5d15029834286b3995273ac3856db P 725fc31fc8b241cb8990f3e8cfb8ac44: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260929 15:55:50.975817 17067 ts_tablet_manager.cc:1905] T 40c5d15029834286b3995273ac3856db P 725fc31fc8b241cb8990f3e8cfb8ac44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:55:50.982908 17067 ts_tablet_manager.cc:1918] T 40c5d15029834286b3995273ac3856db P 725fc31fc8b241cb8990f3e8cfb8ac44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:55:50.983120 17067 log.cc:1196] T 40c5d15029834286b3995273ac3856db P 725fc31fc8b241cb8990f3e8cfb8ac44: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/wal/wals/40c5d15029834286b3995273ac3856db
I20260929 15:55:50.983572 17067 ts_tablet_manager.cc:1939] T 40c5d15029834286b3995273ac3856db P 725fc31fc8b241cb8990f3e8cfb8ac44: Deleting consensus metadata
I20260929 15:55:50.983964 17005 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260929 15:55:50.984103 17005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.001s sys 0.000s
I20260929 15:55:50.984248 17005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:55:50.984361 17005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:55:51.020648 17005 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35363
I20260929 15:55:51.020735 17173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35363 every 8 connection(s)
I20260929 15:55:51.022014 17005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:55:51.025568 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 17005
I20260929 15:55:51.026189 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:37361
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:38491
--webserver_port=33541
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:55:51.026947 17176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:51.024865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.1:58605)
0929 15:55:51.025760 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:51.026022 (+ 262us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":387,"threads_started":1}
W20260929 15:55:51.027542 17174 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:38491 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:38491: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
I20260929 15:55:51.028460 17176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:51.028123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.1:59519)
0929 15:55:51.028281 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:51.028355 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 15:55:51.029474 17176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:51.029061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.1:51935)
0929 15:55:51.029210 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:51.029289 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260929 15:55:51.029739 17174 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 15:55:51.353672 17177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:55:51.353955 17177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:55:51.354049 17177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:55:51.361135 17177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:55:51.361243 17177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:55:51.361371 17177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:55:51.371062 17177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:37361
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=33541
--tserver_master_addrs=127.0.97.62:38491
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:55:51.372267 17177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:55:51.373502 17177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:55:51.386058 17183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:55:51.386127 17185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:55:51.386111 17182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:55:51.387287 17177 server_base.cc:1034] running on GCE node
I20260929 15:55:51.387949 17177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:55:51.388965 17177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:55:51.390213 17177 hybrid_clock.cc:648] HybridClock initialized: now 1790697351390146 us; error 58 us; skew 500 ppm
I20260929 15:55:51.390564 17177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:55:51 dist-test-slave-r0ck krb5kdc[15856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:55:51.393862 17177 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:55:51.395481 17177 webserver.cc:460] Webserver started at http://127.0.97.2:33541/ using document root <none> and password file <none>
I20260929 15:55:51.396181 17177 fs_manager.cc:359] Metadata directory not provided
I20260929 15:55:51.396328 17177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:55:51.401762 17177 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:55:51.404524 17192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:55:51.405866 17177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:55:51.406036 17177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/wal
uuid: "b2cce6a0fc254095826871e8d5a0a0fc"
format_stamp: "Formatted at 2026-09-29 15:55:05 on dist-test-slave-r0ck"
server_key: "724f69044ce0f997bb65f206403f49ba"
server_key_iv: "877f61eb7d1787f7c87f742607d0b2ac"
server_key_version: "encryptionkey@0"
I20260929 15:55:51.407002 17177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:55:51.444365 17177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:55:51.448426 17177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:55:51.448693 17177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:55:51.449700 17177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:55:51.451076 17177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:55:51.451200 17177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:55:51.451326 17177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:55:51.451409 17177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:55:51.485319 17177 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37361
I20260929 15:55:51.485471 17304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37361 every 8 connection(s)
I20260929 15:55:51.486755 17177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:55:51.489778 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 17177
I20260929 15:55:51.490559 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:39479
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:38491
--webserver_port=34951
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:55:51.491590 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:51.489686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.2:43673)
0929 15:55:51.490484 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:51.490726 (+ 242us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":307,"threads_started":1}
W20260929 15:55:51.492293 17305 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:38491 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:38491: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
I20260929 15:55:51.493114 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:51.492792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.2:53629)
0929 15:55:51.492940 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:51.493015 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 15:55:51.494058 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:51.493631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.2:38309)
0929 15:55:51.493800 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:51.493899 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 15:55:51.494355 17305 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 15:55:51.824446 17308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:55:51.824733 17308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:55:51.824831 17308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:55:51.832511 17308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:55:51.832629 17308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:55:51.832782 17308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:55:51.842576 17308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:39479
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=34951
--tserver_master_addrs=127.0.97.62:38491
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:55:51.843888 17308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:55:51.845117 17308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:55:51.858366 17313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:55:51.858306 17314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:55:51.858392 17316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:55:51.859642 17308 server_base.cc:1034] running on GCE node
I20260929 15:55:51.860291 17308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:55:51.861279 17308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:55:51.862522 17308 hybrid_clock.cc:648] HybridClock initialized: now 1790697351862479 us; error 38 us; skew 500 ppm
I20260929 15:55:51.862847 17308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:55:51 dist-test-slave-r0ck krb5kdc[15856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:55:51.865981 17308 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:55:51.867552 17308 webserver.cc:460] Webserver started at http://127.0.97.3:34951/ using document root <none> and password file <none>
I20260929 15:55:51.868266 17308 fs_manager.cc:359] Metadata directory not provided
I20260929 15:55:51.868422 17308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:55:51.874102 17308 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 15:55:51.876955 17323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:55:51.878435 17308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 15:55:51.878633 17308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/wal
uuid: "129d2804e3d54d01ad6eef1c52714dc9"
format_stamp: "Formatted at 2026-09-29 15:55:06 on dist-test-slave-r0ck"
server_key: "2305eba166af36f9aa4e4e048b2da4f6"
server_key_iv: "6546181ab497dccfbdc51021702bded9"
server_key_version: "encryptionkey@0"
I20260929 15:55:51.879644 17308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:55:51.898166 17308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:55:51.902755 17308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:55:51.903043 17308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:55:51.904213 17308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:55:51.905640 17308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:55:51.905768 17308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:55:51.905907 17308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:55:51.905997 17308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:55:51.940055 17308 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39479
I20260929 15:55:51.940109 17435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39479 every 8 connection(s)
I20260929 15:55:51.941545 17308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:55:51.944372 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 17308
I20260929 15:55:51.946759 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:51.944711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.3:34243)
0929 15:55:51.945599 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:51.945858 (+ 259us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":345,"threads_started":1}
W20260929 15:55:51.947461 17436 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:38491 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:38491: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
I20260929 15:55:51.948215 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:51.947895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.3:49759)
0929 15:55:51.948037 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:51.948104 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 15:55:51.948995 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:51.948683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.3:43899)
0929 15:55:51.948822 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:51.948896 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 15:55:51.949218 17436 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 15:55:52.031018 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:52.030267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.1:60081)
0929 15:55:52.030701 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:52.030790 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":208,"threads_started":1}
I20260929 15:55:52.495603 17441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:52.494873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.2:53955)
0929 15:55:52.495299 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:52.495385 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":210,"threads_started":1}
I20260929 15:55:52.950692 17442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:52.949790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.3:58539)
0929 15:55:52.950312 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:52.950410 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":255,"threads_started":1}
I20260929 15:55:53.032579 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:53.031894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.1:35731)
0929 15:55:53.032331 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:53.032409 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":226,"threads_started":1}
I20260929 15:55:53.497190 17444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:53.496499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.2:60095)
0929 15:55:53.496942 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:53.497030 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":230,"threads_started":1}
I20260929 15:55:53.952746 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:53.951957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.3:35185)
0929 15:55:53.952490 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:53.952584 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":301,"threads_started":1}
I20260929 15:55:54.034252 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:54.033492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.1:38533)
0929 15:55:54.033984 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:54.034076 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":242,"threads_started":1}
I20260929 15:55:54.498770 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:54.498078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.2:35855)
0929 15:55:54.498521 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:54.498601 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":237,"threads_started":1}
I20260929 15:55:54.954442 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:54.953809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.3:51741)
0929 15:55:54.954198 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:54.954273 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":199,"threads_started":1}
I20260929 15:55:55.035950 17449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:55.035190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.1:60967)
0929 15:55:55.035697 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:55.035780 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":300,"threads_started":1}
I20260929 15:55:55.500387 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:55.499667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.2:41257)
0929 15:55:55.500137 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:55.500216 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":230,"threads_started":1}
I20260929 15:55:55.955991 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:55.955295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.3:46353)
0929 15:55:55.955708 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:55.955801 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":215,"threads_started":1}
I20260929 15:55:56.037755 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:56.036967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.1:35191)
0929 15:55:56.037482 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:56.037568 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":225,"threads_started":1}
I20260929 15:55:56.502183 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:56.501441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.2:51047)
0929 15:55:56.501907 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:56.502012 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":245,"threads_started":1}
I20260929 15:55:56.957950 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:56.957010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.3:36939)
0929 15:55:56.957679 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:56.957767 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":384,"threads_started":1}
I20260929 15:55:57.039356 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:57.038632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.1:55219)
0929 15:55:57.039076 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:57.039200 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":231,"threads_started":1}
I20260929 15:55:57.503736 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:57.503066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.2:51259)
0929 15:55:57.503482 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:57.503568 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":220,"threads_started":1}
I20260929 15:55:57.959810 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:57.958974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.3:47555)
0929 15:55:57.959544 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:57.959632 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":306,"threads_started":1}
I20260929 15:55:58.041056 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:58.040209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.1:37655)
0929 15:55:58.040727 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:58.040811 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":242,"threads_started":1}
I20260929 15:55:58.505609 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:58.504750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.2:33993)
0929 15:55:58.505242 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:58.505323 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":222,"threads_started":1}
I20260929 15:55:58.961560 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:58.960689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.3:54161)
0929 15:55:58.961188 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:58.961270 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":228,"threads_started":1}
I20260929 15:55:59.042721 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:59.041986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.1:54547)
0929 15:55:59.042472 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:59.042549 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":227,"threads_started":1}
I20260929 15:55:59.507121 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:59.506472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.2:60283)
0929 15:55:59.506892 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:59.506966 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":230,"threads_started":1}
I20260929 15:55:59.963061 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:55:59.962401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.3:57831)
0929 15:55:59.962826 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0929 15:55:59.962905 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":229,"threads_started":1}
I20260929 15:56:00.049130 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:00.048086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.1:45315)
0929 15:56:00.048713 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:00.048882 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":329,"threads_started":1}
I20260929 15:56:00.508960 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:00.508107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.2:50741)
0929 15:56:00.508662 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:00.508747 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":307,"threads_started":1}
I20260929 15:56:00.964742 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:00.964029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.3:36295)
0929 15:56:00.964489 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:00.964575 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":230,"threads_started":1}
I20260929 15:56:01.051005 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:01.050190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.1:47643)
0929 15:56:01.050764 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:01.050841 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":303,"threads_started":1}
I20260929 15:56:01.510715 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:01.509990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.2:35187)
0929 15:56:01.510474 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:01.510553 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":287,"threads_started":1}
I20260929 15:56:01.966567 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:01.965857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.3:53849)
0929 15:56:01.966317 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:01.966405 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":252,"threads_started":1}
I20260929 15:56:02.052675 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:02.051919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.1:57199)
0929 15:56:02.052414 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:02.052490 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":261,"threads_started":1}
I20260929 15:56:02.512348 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:02.511614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.2:49149)
0929 15:56:02.512080 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:02.512165 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":264,"threads_started":1}
I20260929 15:56:02.968142 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:02.967464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.3:38837)
0929 15:56:02.967887 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:02.967971 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":223,"threads_started":1}
I20260929 15:56:03.054489 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:03.053713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.1:33987)
0929 15:56:03.054224 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:03.054330 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":277,"threads_started":1}
I20260929 15:56:03.514082 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:03.513341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.2:46773)
0929 15:56:03.513839 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:03.513922 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":209,"threads_started":1}
I20260929 15:56:03.969749 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:03.969023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.3:59605)
0929 15:56:03.969515 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:03.969598 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":221,"threads_started":1}
I20260929 15:56:04.056051 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:04.055303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.1:34891)
0929 15:56:04.055755 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:04.055846 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":235,"threads_started":1}
I20260929 15:56:04.515760 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:04.515011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.2:48799)
0929 15:56:04.515474 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:04.515563 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":246,"threads_started":1}
I20260929 15:56:04.971342 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:04.970622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.3:42837)
0929 15:56:04.971076 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:04.971160 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":231,"threads_started":1}
I20260929 15:56:05.057787 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:05.057014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.1:58023)
0929 15:56:05.057528 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:05.057607 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":226,"threads_started":1}
I20260929 15:56:05.517563 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:05.516760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.2:54187)
0929 15:56:05.517211 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:05.517289 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":220,"threads_started":1}
I20260929 15:56:05.972905 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:05.972180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.3:50005)
0929 15:56:05.972641 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:05.972728 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":230,"threads_started":1}
I20260929 15:56:06.059517 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:06.058828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.1:57321)
0929 15:56:06.059268 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:06.059341 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":239,"threads_started":1}
I20260929 15:56:06.519378 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:06.518692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.2:36811)
0929 15:56:06.519108 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:06.519182 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":208,"threads_started":1}
I20260929 15:56:06.949618 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 17005
I20260929 15:56:06.974175 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:06.950871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53072 (local address 127.0.97.1:35363)
0929 15:56:06.951442 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:06.951456 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:06.951492 (+ 36us) server_negotiation.cc:407] Connection header received
0929 15:56:06.953156 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:06.953166 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:06.953690 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:06.953974 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:06.955364 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:06.956598 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:06.957840 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:06.958490 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:06.958625 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:06.967473 (+ 8848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:06.967483 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:06.967501 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:56:06.967546 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:06.970911 (+ 3365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:06.971604 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:06.971612 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:06.971618 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:06.971799 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:06.972173 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:06.972177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:06.972179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:06.972718 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:06.972950 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:06.973419 (+ 469us) server_negotiation.cc:299] Negotiation successful
0929 15:56:06.973864 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":319,"threads_started":1}
I20260929 15:56:06.976065 17169 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:56:06.976387 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:06.974114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.3:60571)
0929 15:56:06.976088 (+ 1974us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:06.976167 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1847,"thread_start_us":1753,"threads_started":1}
I20260929 15:56:07.061820 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:07.060649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38491 (local address 127.0.97.1:38833)
0929 15:56:07.061221 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:07.061426 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":302,"threads_started":1}
I20260929 15:56:07.061942 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 17005
I20260929 15:56:07.080968 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 17177
I20260929 15:56:07.097920 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:07.081956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46044 (local address 127.0.97.2:37361)
0929 15:56:07.082430 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:07.082444 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:07.082481 (+ 37us) server_negotiation.cc:407] Connection header received
0929 15:56:07.084319 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:07.084329 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:07.084735 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:07.085021 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:07.086493 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:07.087691 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:07.088952 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:07.089656 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:07.089792 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:07.092252 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:07.092259 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:07.092274 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:56:07.092309 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:07.095195 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:07.095776 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:07.095782 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:07.095787 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:07.095950 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:07.096270 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:07.096273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:07.096275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:07.096718 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:07.096940 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:07.097270 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 15:56:07.097633 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":214,"threads_started":1}
I20260929 15:56:07.099222 17300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:56:07.181335 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 17177
I20260929 15:56:07.199115 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 17308
I20260929 15:56:07.216178 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:07.199999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34850 (local address 127.0.97.3:39479)
0929 15:56:07.200612 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:07.200626 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:07.200667 (+ 41us) server_negotiation.cc:407] Connection header received
0929 15:56:07.202328 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:07.202338 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:07.202745 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:07.203044 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:07.204302 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:07.205566 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:07.206778 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:07.207415 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:07.207542 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:07.210105 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:07.210112 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:07.210126 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:56:07.210165 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:07.213017 (+ 2852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:07.213684 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:07.213691 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:07.213697 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:07.213859 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:07.214188 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:07.214191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:07.214193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:07.214706 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:07.214933 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:07.215260 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 15:56:07.215863 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":296,"threads_started":1}
I20260929 15:56:07.217584 17431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:56:07.301155 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 17308
2026-09-29T15:56:07Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64199 ms)
[----------] 30 tests from SecurityITest (291594 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:56:07 dist-test-slave-r0ck krb5kdc[17494](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:56:07 dist-test-slave-r0ck krb5kdc[17494](info): set up 2 sockets
Sep 29 15:56:07 dist-test-slave-r0ck krb5kdc[17494](info): commencing operation
krb5kdc: starting...
W20260929 15:56:08.409590 388 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:56:08 dist-test-slave-r0ck krb5kdc[17494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:56:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:56:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:56:08.681248 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35257
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35257
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:56:08.999220 17507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:56:08.999552 17507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:56:08.999728 17507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:56:09.006850 17507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:56:09.007011 17507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:56:09.007124 17507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:56:09.007230 17507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:56:09.007318 17507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:56:09.017165 17507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:56:09.018592 17507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:56:09.019899 17507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:56:09.031160 17512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:09.031216 17513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:09.031206 17515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:56:09.032404 17507 server_base.cc:1034] running on GCE node
I20260929 15:56:09.033062 17507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:56:09.034440 17507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:56:09.035666 17507 hybrid_clock.cc:648] HybridClock initialized: now 1790697369035622 us; error 31 us; skew 500 ppm
I20260929 15:56:09.035974 17507 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:56:09 dist-test-slave-r0ck krb5kdc[17494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:56:09.039160 17507 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:56:09.040719 17507 webserver.cc:460] Webserver started at http://127.0.97.62:33465/ using document root <none> and password file <none>
I20260929 15:56:09.041469 17507 fs_manager.cc:359] Metadata directory not provided
I20260929 15:56:09.041597 17507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:56:09.041939 17507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:56:09.044879 17507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "c74e657f349a4dccb341f46a38577015"
format_stamp: "Formatted at 2026-09-29 15:56:09 on dist-test-slave-r0ck"
server_key: "ff3d16e81b7da857c65ef1fbedda36cc"
server_key_iv: "e8531815dd594aa4ca45d19061f26413"
server_key_version: "encryptionkey@0"
I20260929 15:56:09.045682 17507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "c74e657f349a4dccb341f46a38577015"
format_stamp: "Formatted at 2026-09-29 15:56:09 on dist-test-slave-r0ck"
server_key: "ff3d16e81b7da857c65ef1fbedda36cc"
server_key_iv: "e8531815dd594aa4ca45d19061f26413"
server_key_version: "encryptionkey@0"
I20260929 15:56:09.050765 17507 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 15:56:09.054100 17522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:09.055358 17507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:56:09.055526 17507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "c74e657f349a4dccb341f46a38577015"
format_stamp: "Formatted at 2026-09-29 15:56:09 on dist-test-slave-r0ck"
server_key: "ff3d16e81b7da857c65ef1fbedda36cc"
server_key_iv: "e8531815dd594aa4ca45d19061f26413"
server_key_version: "encryptionkey@0"
I20260929 15:56:09.055720 17507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:56:09.078856 17507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:56:09.083154 17507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:56:09.083492 17507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:56:09.102020 17507 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35257
I20260929 15:56:09.102051 17573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35257 every 8 connection(s)
I20260929 15:56:09.103390 17507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:56:09.106922 17574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:56:09.112875 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 17507
I20260929 15:56:09.113102 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:56:09.113771 388 external_mini_cluster.cc:1428] Setting key d5173cc23157827dec74dbd1c7f01ce6
I20260929 15:56:09.117547 17574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c74e657f349a4dccb341f46a38577015: Bootstrap starting.
I20260929 15:56:09.121135 17574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c74e657f349a4dccb341f46a38577015: Neither blocks nor log segments found. Creating new log.
I20260929 15:56:09.122196 17574 log.cc:824] T 00000000000000000000000000000000 P c74e657f349a4dccb341f46a38577015: Log is configured to *not* fsync() on all Append() calls
I20260929 15:56:09.125003 17574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c74e657f349a4dccb341f46a38577015: No bootstrap required, opened a new log
Sep 29 15:56:09 dist-test-slave-r0ck krb5kdc[17494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:56:09.133780 17574 raft_consensus.cc:348] T 00000000000000000000000000000000 P c74e657f349a4dccb341f46a38577015 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c74e657f349a4dccb341f46a38577015" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35257 } }
I20260929 15:56:09.134177 17574 raft_consensus.cc:374] T 00000000000000000000000000000000 P c74e657f349a4dccb341f46a38577015 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:56:09.134306 17574 raft_consensus.cc:729] T 00000000000000000000000000000000 P c74e657f349a4dccb341f46a38577015 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c74e657f349a4dccb341f46a38577015, State: Initialized, Role: FOLLOWER
I20260929 15:56:09.134935 17574 consensus_queue.cc:260] T 00000000000000000000000000000000 P c74e657f349a4dccb341f46a38577015 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c74e657f349a4dccb341f46a38577015" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35257 } }
I20260929 15:56:09.135177 17574 raft_consensus.cc:388] T 00000000000000000000000000000000 P c74e657f349a4dccb341f46a38577015 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:56:09.135315 17574 raft_consensus.cc:482] T 00000000000000000000000000000000 P c74e657f349a4dccb341f46a38577015 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:56:09.135509 17574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c74e657f349a4dccb341f46a38577015 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:56:09.136797 17577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:09.115791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59690 (local address 127.0.97.62:35257)
0929 15:56:09.116556 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:09.116567 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:09.116604 (+ 37us) server_negotiation.cc:407] Connection header received
0929 15:56:09.118384 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:09.118405 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:09.118757 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:09.119075 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:09.120933 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:09.122408 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:09.124137 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:09.124767 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:09.124912 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:09.129583 (+ 4671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:09.129595 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:09.129609 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:56:09.129646 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:09.132682 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:09.133433 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:09.133442 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:09.133456 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:09.133614 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:09.133992 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:09.133998 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:09.134006 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:09.134658 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:09.134996 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:09.135407 (+ 411us) server_negotiation.cc:299] Negotiation successful
0929 15:56:09.135809 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":371,"threads_started":1}
I20260929 15:56:09.138118 17574 raft_consensus.cc:504] T 00000000000000000000000000000000 P c74e657f349a4dccb341f46a38577015 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c74e657f349a4dccb341f46a38577015" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35257 } }
I20260929 15:56:09.138666 17574 leader_election.cc:302] T 00000000000000000000000000000000 P c74e657f349a4dccb341f46a38577015 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c74e657f349a4dccb341f46a38577015; no voters:
I20260929 15:56:09.139293 17574 leader_election.cc:288] T 00000000000000000000000000000000 P c74e657f349a4dccb341f46a38577015 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:56:09.139427 17579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c74e657f349a4dccb341f46a38577015 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:56:09.140057 17579 raft_consensus.cc:686] T 00000000000000000000000000000000 P c74e657f349a4dccb341f46a38577015 [term 1 LEADER]: Becoming Leader. State: Replica: c74e657f349a4dccb341f46a38577015, State: Running, Role: LEADER
I20260929 15:56:09.140717 17579 consensus_queue.cc:237] T 00000000000000000000000000000000 P c74e657f349a4dccb341f46a38577015 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c74e657f349a4dccb341f46a38577015" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35257 } }
I20260929 15:56:09.141846 17574 sys_catalog.cc:564] T 00000000000000000000000000000000 P c74e657f349a4dccb341f46a38577015 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:56:09.143476 17581 sys_catalog.cc:455] T 00000000000000000000000000000000 P c74e657f349a4dccb341f46a38577015 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c74e657f349a4dccb341f46a38577015" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c74e657f349a4dccb341f46a38577015" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35257 } } }
I20260929 15:56:09.143831 17581 sys_catalog.cc:458] T 00000000000000000000000000000000 P c74e657f349a4dccb341f46a38577015 [sys.catalog]: This master's current role is: LEADER
I20260929 15:56:09.144286 17580 sys_catalog.cc:455] T 00000000000000000000000000000000 P c74e657f349a4dccb341f46a38577015 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c74e657f349a4dccb341f46a38577015. Latest consensus state: current_term: 1 leader_uuid: "c74e657f349a4dccb341f46a38577015" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c74e657f349a4dccb341f46a38577015" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35257 } } }
I20260929 15:56:09.144594 17580 sys_catalog.cc:458] T 00000000000000000000000000000000 P c74e657f349a4dccb341f46a38577015 [sys.catalog]: This master's current role is: LEADER
I20260929 15:56:09.150080 17585 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:56:09.154224 17585 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:56:09.162691 17585 catalog_manager.cc:1347] Generated new cluster ID: de25392a6df746b3b8a2dc6a61788476
I20260929 15:56:09.162822 17585 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:56:09.177876 17585 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:56:09.179137 17585 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:56:09.190297 17585 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c74e657f349a4dccb341f46a38577015: Generated new TSK 0
I20260929 15:56:09.191166 17585 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 15:56:09.252957 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35257
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:56:09.572762 17600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:56:09.573026 17600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:56:09.573184 17600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:56:09.580330 17600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:56:09.580484 17600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:56:09.580670 17600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:56:09.590193 17600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:56:09.591466 17600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:56:09.592819 17600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:56:09.605393 17605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:09.605378 17608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:09.605473 17606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:56:09.606093 17600 server_base.cc:1034] running on GCE node
I20260929 15:56:09.606644 17600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:56:09.607511 17600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:56:09.608736 17600 hybrid_clock.cc:648] HybridClock initialized: now 1790697369608696 us; error 53 us; skew 500 ppm
I20260929 15:56:09.609046 17600 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:56:09 dist-test-slave-r0ck krb5kdc[17494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:56:09.612221 17600 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:56:09.613689 17600 webserver.cc:460] Webserver started at http://127.0.97.1:43209/ using document root <none> and password file <none>
I20260929 15:56:09.614382 17600 fs_manager.cc:359] Metadata directory not provided
I20260929 15:56:09.614523 17600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:56:09.614830 17600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:56:09.617697 17600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-0/data/instance:
uuid: "cc75fb1ec51b40778f2d150548d45835"
format_stamp: "Formatted at 2026-09-29 15:56:09 on dist-test-slave-r0ck"
server_key: "238de20576929b3b15118a7eb15548b0"
server_key_iv: "63ce59b9916b492880551d51bad7a058"
server_key_version: "encryptionkey@0"
I20260929 15:56:09.618462 17600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "cc75fb1ec51b40778f2d150548d45835"
format_stamp: "Formatted at 2026-09-29 15:56:09 on dist-test-slave-r0ck"
server_key: "238de20576929b3b15118a7eb15548b0"
server_key_iv: "63ce59b9916b492880551d51bad7a058"
server_key_version: "encryptionkey@0"
I20260929 15:56:09.623281 17600 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 15:56:09.626705 17615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:09.627959 17600 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:56:09.628141 17600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-0/wal
uuid: "cc75fb1ec51b40778f2d150548d45835"
format_stamp: "Formatted at 2026-09-29 15:56:09 on dist-test-slave-r0ck"
server_key: "238de20576929b3b15118a7eb15548b0"
server_key_iv: "63ce59b9916b492880551d51bad7a058"
server_key_version: "encryptionkey@0"
I20260929 15:56:09.628353 17600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:56:09.651409 17600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:56:09.655725 17600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:56:09.656018 17600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:56:09.656961 17600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:56:09.658317 17600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:56:09.658417 17600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:09.658564 17600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:56:09.658638 17600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:09.691637 17600 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41129
I20260929 15:56:09.691727 17727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41129 every 8 connection(s)
I20260929 15:56:09.692893 17600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:56:09.695449 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 17600
I20260929 15:56:09.695684 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:56:09.696516 388 external_mini_cluster.cc:1428] Setting key 09a7c82f5cb8b1113f3ba0549b7f629a
Sep 29 15:56:09 dist-test-slave-r0ck krb5kdc[17494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:56:09.712435 17730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:09.695495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42767 (local address 127.0.97.62:35257)
0929 15:56:09.695962 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:09.695967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:09.697068 (+ 1101us) server_negotiation.cc:407] Connection header received
0929 15:56:09.699092 (+ 2024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:09.699099 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:09.699199 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:09.699323 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:09.700712 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:09.701524 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:09.703006 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:09.703414 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:09.703557 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:09.706946 (+ 3389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:09.706952 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:09.706955 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:56:09.706985 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:09.709516 (+ 2531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:09.710202 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:09.710209 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:09.710212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:09.710341 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:09.710770 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:09.710774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:09.710778 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:09.711034 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:09.711197 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:09.711843 (+ 646us) server_negotiation.cc:299] Negotiation successful
0929 15:56:09.712060 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":244,"threads_started":1}
I20260929 15:56:09.713578 17731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:09.696129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35257 (local address 127.0.97.1:42767)
0929 15:56:09.696917 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:09.696953 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:09.698842 (+ 1889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:09.699534 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:09.699542 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:09.700039 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:56:09.700549 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:09.700565 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:09.701679 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:09.701683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:09.702863 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:09.702874 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:09.703644 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:09.703653 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:09.703898 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:09.704571 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:56:09.704593 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:56:09.706730 (+ 2137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:56:09.709651 (+ 2921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:56:09.709659 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:56:09.709675 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:56:09.710083 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:56:09.710461 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:56:09.710465 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:56:09.710469 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:56:09.710650 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:56:09.711172 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:56:09.711179 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:56:09.711493 (+ 314us) client_negotiation.cc:769] Sending connection context
0929 15:56:09.711749 (+ 256us) client_negotiation.cc:240] Negotiation successful
0929 15:56:09.712250 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":326,"threads_started":1}
I20260929 15:56:09.715050 17728 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35257
I20260929 15:56:09.715394 17728 heartbeater.cc:464] Registering TS with master...
I20260929 15:56:09.716311 17728 heartbeater.cc:511] Master 127.0.97.62:35257 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 15:56:09.718502 17539 ts_manager.cc:194] Registered new tserver with Master: cc75fb1ec51b40778f2d150548d45835 (127.0.97.1:41129)
I20260929 15:56:09.720052 17539 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42767
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 15:56:09.748677 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35257
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:56:10.077229 17734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:56:10.077510 17734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:56:10.077698 17734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:56:10.084869 17734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:56:10.084975 17734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:56:10.085182 17734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:56:10.094901 17734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:56:10.096161 17734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:56:10.097548 17734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:56:10.110395 17739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:10.110333 17742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:10.110247 17740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:56:10.111573 17734 server_base.cc:1034] running on GCE node
I20260929 15:56:10.112171 17734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:56:10.113162 17734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:56:10.114399 17734 hybrid_clock.cc:648] HybridClock initialized: now 1790697370114341 us; error 53 us; skew 500 ppm
I20260929 15:56:10.114709 17734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:56:10 dist-test-slave-r0ck krb5kdc[17494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697370, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:56:10.118022 17734 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:56:10.119542 17734 webserver.cc:460] Webserver started at http://127.0.97.2:36163/ using document root <none> and password file <none>
I20260929 15:56:10.120251 17734 fs_manager.cc:359] Metadata directory not provided
I20260929 15:56:10.120386 17734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:56:10.120698 17734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:56:10.123603 17734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-1/data/instance:
uuid: "59ed1556867d4db3a023c7b887ad11cd"
format_stamp: "Formatted at 2026-09-29 15:56:10 on dist-test-slave-r0ck"
server_key: "c59e458158a36ad2de75d729be044551"
server_key_iv: "16e66ba5728e5f55c6c0fc8586a7e5b0"
server_key_version: "encryptionkey@0"
I20260929 15:56:10.124397 17734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "59ed1556867d4db3a023c7b887ad11cd"
format_stamp: "Formatted at 2026-09-29 15:56:10 on dist-test-slave-r0ck"
server_key: "c59e458158a36ad2de75d729be044551"
server_key_iv: "16e66ba5728e5f55c6c0fc8586a7e5b0"
server_key_version: "encryptionkey@0"
I20260929 15:56:10.129580 17734 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 15:56:10.133060 17749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:10.134419 17734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:56:10.134594 17734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-1/wal
uuid: "59ed1556867d4db3a023c7b887ad11cd"
format_stamp: "Formatted at 2026-09-29 15:56:10 on dist-test-slave-r0ck"
server_key: "c59e458158a36ad2de75d729be044551"
server_key_iv: "16e66ba5728e5f55c6c0fc8586a7e5b0"
server_key_version: "encryptionkey@0"
I20260929 15:56:10.134792 17734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:56:10.150487 17734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:56:10.154690 17734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:56:10.154928 17734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:56:10.155849 17734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:56:10.157141 17734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:56:10.157214 17734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:10.157296 17734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:56:10.157398 17734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:10.190392 17734 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40389
I20260929 15:56:10.190485 17861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40389 every 8 connection(s)
I20260929 15:56:10.191673 17734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:56:10.201845 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 17734
I20260929 15:56:10.202035 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:56:10.202576 388 external_mini_cluster.cc:1428] Setting key efb46fab728940f8f45ffd03942e6f7b
Sep 29 15:56:10 dist-test-slave-r0ck krb5kdc[17494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697370, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:56:10.212885 17730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:10.194292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33221 (local address 127.0.97.62:35257)
0929 15:56:10.194557 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:10.194565 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:10.195677 (+ 1112us) server_negotiation.cc:407] Connection header received
0929 15:56:10.197451 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:10.197455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:10.197525 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:10.197614 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:10.199079 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:10.199773 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:10.201315 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:10.202079 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:10.202209 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:10.205881 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:10.205890 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:10.205898 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:56:10.205945 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:10.209615 (+ 3670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:10.210325 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:10.210334 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:10.210341 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:10.210508 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:10.210953 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:10.210960 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:10.210966 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:10.211326 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:10.211476 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:10.212173 (+ 697us) server_negotiation.cc:299] Negotiation successful
0929 15:56:10.212444 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260929 15:56:10.213647 17864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:10.194747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35257 (local address 127.0.97.2:33221)
0929 15:56:10.195499 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:10.195548 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:10.197136 (+ 1588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:10.197815 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:10.197822 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:10.198365 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:56:10.198909 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:10.198926 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:10.199909 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:10.199912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:10.201134 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:10.201145 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:10.202326 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:10.202331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:10.202522 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:10.203271 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:56:10.203292 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:56:10.205644 (+ 2352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:56:10.209758 (+ 4114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:56:10.209766 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:56:10.209777 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:56:10.210203 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:56:10.210635 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:56:10.210639 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:56:10.210642 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:56:10.210839 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:56:10.211483 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:56:10.211489 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:56:10.211812 (+ 323us) client_negotiation.cc:769] Sending connection context
0929 15:56:10.212035 (+ 223us) client_negotiation.cc:240] Negotiation successful
0929 15:56:10.212430 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":300,"threads_started":1}
I20260929 15:56:10.215059 17862 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35257
I20260929 15:56:10.215368 17862 heartbeater.cc:464] Registering TS with master...
I20260929 15:56:10.216184 17862 heartbeater.cc:511] Master 127.0.97.62:35257 requested a full tablet report, sending...
I20260929 15:56:10.217689 17539 ts_manager.cc:194] Registered new tserver with Master: 59ed1556867d4db3a023c7b887ad11cd (127.0.97.2:40389)
I20260929 15:56:10.218501 17539 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33221
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 15:56:10.260684 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35257
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:56:10.603332 17867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:56:10.603585 17867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:56:10.603744 17867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:56:10.610965 17867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:56:10.611088 17867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:56:10.611258 17867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:56:10.621008 17867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:56:10.622339 17867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:56:10.623603 17867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:56:10.636559 17872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:10.636618 17873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:10.636732 17875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:56:10.637545 17867 server_base.cc:1034] running on GCE node
I20260929 15:56:10.638108 17867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:56:10.639000 17867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:56:10.640197 17867 hybrid_clock.cc:648] HybridClock initialized: now 1790697370640160 us; error 37 us; skew 500 ppm
I20260929 15:56:10.640511 17867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:56:10 dist-test-slave-r0ck krb5kdc[17494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697370, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:56:10.643811 17867 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:56:10.645193 17867 webserver.cc:460] Webserver started at http://127.0.97.3:39269/ using document root <none> and password file <none>
I20260929 15:56:10.645884 17867 fs_manager.cc:359] Metadata directory not provided
I20260929 15:56:10.646010 17867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:56:10.646308 17867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:56:10.649073 17867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-2/data/instance:
uuid: "348ab908d1ac40faad1950ca9ad90571"
format_stamp: "Formatted at 2026-09-29 15:56:10 on dist-test-slave-r0ck"
server_key: "00e317df100c17787cadbdffaa50bbeb"
server_key_iv: "c7cf0607491b112768aa2f177a06b143"
server_key_version: "encryptionkey@0"
I20260929 15:56:10.649901 17867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "348ab908d1ac40faad1950ca9ad90571"
format_stamp: "Formatted at 2026-09-29 15:56:10 on dist-test-slave-r0ck"
server_key: "00e317df100c17787cadbdffaa50bbeb"
server_key_iv: "c7cf0607491b112768aa2f177a06b143"
server_key_version: "encryptionkey@0"
I20260929 15:56:10.654897 17867 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 15:56:10.658044 17882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:10.659262 17867 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 15:56:10.659416 17867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-2/wal
uuid: "348ab908d1ac40faad1950ca9ad90571"
format_stamp: "Formatted at 2026-09-29 15:56:10 on dist-test-slave-r0ck"
server_key: "00e317df100c17787cadbdffaa50bbeb"
server_key_iv: "c7cf0607491b112768aa2f177a06b143"
server_key_version: "encryptionkey@0"
I20260929 15:56:10.659650 17867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:56:10.676434 17867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:56:10.680488 17867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:56:10.680716 17867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:56:10.681583 17867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:56:10.682811 17867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:56:10.682879 17867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:10.683002 17867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:56:10.683076 17867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:10.717063 17867 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33799
I20260929 15:56:10.717101 17994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33799 every 8 connection(s)
I20260929 15:56:10.718286 17867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:56:10.722851 17728 heartbeater.cc:503] Master 127.0.97.62:35257 was elected leader, sending a full tablet report...
I20260929 15:56:10.723825 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 17867
I20260929 15:56:10.724009 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:56:10.724581 388 external_mini_cluster.cc:1428] Setting key 2ac93df53a263d52568797d5807a91c1
Sep 29 15:56:10 dist-test-slave-r0ck krb5kdc[17494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697370, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:56:10.737090 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:10.720710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50529 (local address 127.0.97.62:35257)
0929 15:56:10.721156 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:10.721160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:10.722111 (+ 951us) server_negotiation.cc:407] Connection header received
0929 15:56:10.724191 (+ 2080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:10.724198 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:10.724293 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:10.724406 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:10.726083 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:10.727096 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:10.728603 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:10.728972 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:10.729066 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:10.732294 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:10.732298 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:10.732301 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:56:10.732330 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:10.734706 (+ 2376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:10.735317 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:10.735320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:10.735322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:10.735432 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:10.735770 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:10.735773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:10.735775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:10.736001 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:10.736111 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:10.736695 (+ 584us) server_negotiation.cc:299] Negotiation successful
0929 15:56:10.736842 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":223,"threads_started":1}
I20260929 15:56:10.737998 17998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:10.721223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35257 (local address 127.0.97.3:50529)
0929 15:56:10.721961 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:10.722002 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:10.723871 (+ 1869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:10.724694 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:10.724706 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:10.725253 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:56:10.725921 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:10.725939 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:10.727252 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:10.727261 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:10.728477 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:10.728488 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:10.729148 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:10.729153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:10.729418 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:10.729996 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:56:10.730020 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:56:10.732110 (+ 2090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:56:10.734823 (+ 2713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:56:10.734829 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:56:10.734839 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:56:10.735212 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:56:10.735517 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:56:10.735520 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:56:10.735522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:56:10.735677 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:56:10.736099 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:56:10.736104 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:56:10.736400 (+ 296us) client_negotiation.cc:769] Sending connection context
0929 15:56:10.736600 (+ 200us) client_negotiation.cc:240] Negotiation successful
0929 15:56:10.736917 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":285,"threads_started":1}
I20260929 15:56:10.739461 17995 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35257
I20260929 15:56:10.739737 17995 heartbeater.cc:464] Registering TS with master...
I20260929 15:56:10.740387 17995 heartbeater.cc:511] Master 127.0.97.62:35257 requested a full tablet report, sending...
I20260929 15:56:10.741783 17539 ts_manager.cc:194] Registered new tserver with Master: 348ab908d1ac40faad1950ca9ad90571 (127.0.97.3:33799)
I20260929 15:56:10.742609 17539 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50529
I20260929 15:56:10.751374 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:56:10.768110 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:10.755611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59692 (local address 127.0.97.62:35257)
0929 15:56:10.755778 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:10.755781 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:10.756233 (+ 452us) server_negotiation.cc:407] Connection header received
0929 15:56:10.756358 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:10.756361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:10.756421 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:10.756517 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:10.757924 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:10.758635 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:10.760012 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:10.760490 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:10.760581 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:10.762947 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:10.762951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:10.762954 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:56:10.762982 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:10.765879 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:10.766489 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:10.766493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:10.766496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:10.766606 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:10.766931 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:10.766934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:10.766936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:10.767227 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:10.767430 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:10.767727 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 15:56:10.767887 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 15:56:10.771795 17539 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59692:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:56:10.778475 17539 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:56:10.816974 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:10.789972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35706 (local address 127.0.97.2:40389)
0929 15:56:10.803374 (+ 13402us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:10.803386 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:10.803418 (+ 32us) server_negotiation.cc:407] Connection header received
0929 15:56:10.803518 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:10.803527 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:10.803772 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:10.803949 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:56:10.804482 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:10.807214 (+ 2732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:10.812286 (+ 5072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:10.815582 (+ 3296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:10.815742 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:10.815961 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:10.816453 (+ 492us) server_negotiation.cc:299] Negotiation successful
0929 15:56:10.816618 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13230,"thread_start_us":13118,"threads_started":1}
I20260929 15:56:10.817018 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:10.789710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40389 (local address 127.0.0.1:35706)
0929 15:56:10.802532 (+ 12822us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:10.802572 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:10.802772 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:10.803957 (+ 1185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:10.803965 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:10.803991 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:56:10.804311 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:10.804322 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:10.807425 (+ 3103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:10.807437 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:10.812002 (+ 4565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:10.812022 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:10.816081 (+ 4059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:10.816085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:10.816225 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:10.816258 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 15:56:10.816399 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 15:56:10.816600 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11050,"mutex_wait_us":4,"thread_start_us":13713,"threads_started":1}
I20260929 15:56:10.817811 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:10.789850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41129 (local address 127.0.0.1:53572)
0929 15:56:10.803661 (+ 13811us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:10.803677 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:10.803782 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:10.806834 (+ 3052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:10.806842 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:10.806880 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:56:10.807388 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:10.807404 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:10.810208 (+ 2804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:10.810217 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:10.813269 (+ 3052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:10.813288 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:10.816617 (+ 3329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:10.816626 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:10.816907 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:10.816960 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 15:56:10.817043 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 15:56:10.817224 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13701,"mutex_wait_us":52,"thread_start_us":11199,"threads_started":1}
I20260929 15:56:10.819054 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:10.790115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53572 (local address 127.0.97.1:41129)
0929 15:56:10.805911 (+ 15796us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:10.805923 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:10.805958 (+ 35us) server_negotiation.cc:407] Connection header received
0929 15:56:10.806075 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:10.806085 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:10.806351 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:10.807637 (+ 1286us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:56:10.807962 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:10.810018 (+ 2056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:10.813536 (+ 3518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:10.815561 (+ 2025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:10.818246 (+ 2685us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:10.818491 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:10.818568 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 15:56:10.818743 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15554,"thread_start_us":16407,"threads_started":1}
I20260929 15:56:10.821224 17797 tablet_service.cc:1467] Processing CreateTablet for tablet c83e112172394012b09a659a2cf10a5d (DEFAULT_TABLE table=test-table [id=9ac2006644f245bd8f76af95955c7e04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:56:10.821945 17663 tablet_service.cc:1467] Processing CreateTablet for tablet c83e112172394012b09a659a2cf10a5d (DEFAULT_TABLE table=test-table [id=9ac2006644f245bd8f76af95955c7e04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:56:10.822620 17797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c83e112172394012b09a659a2cf10a5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:56:10.823207 17663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c83e112172394012b09a659a2cf10a5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:56:10.826525 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:10.805099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33799 (local address 127.0.0.1:39268)
0929 15:56:10.816189 (+ 11090us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:10.816210 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:10.816343 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:10.819568 (+ 3225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:10.819576 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:10.819604 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:56:10.819906 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:10.819914 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:10.821891 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:10.821896 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:10.823906 (+ 2010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:10.823920 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:10.825772 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:10.825778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:10.825943 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:10.825976 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 15:56:10.826074 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 15:56:10.826203 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10912,"thread_start_us":10743,"threads_started":1}
I20260929 15:56:10.826623 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:10.804249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39268 (local address 127.0.97.3:33799)
0929 15:56:10.818967 (+ 14718us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:10.818975 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:10.819006 (+ 31us) server_negotiation.cc:407] Connection header received
0929 15:56:10.819104 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:10.819112 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:10.819349 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:10.819528 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:56:10.820111 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:10.821740 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:10.824103 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:10.825551 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:10.825680 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:10.825889 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:10.826130 (+ 241us) server_negotiation.cc:299] Negotiation successful
0929 15:56:10.826283 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14519,"thread_start_us":15058,"threads_started":1}
I20260929 15:56:10.830317 17930 tablet_service.cc:1467] Processing CreateTablet for tablet c83e112172394012b09a659a2cf10a5d (DEFAULT_TABLE table=test-table [id=9ac2006644f245bd8f76af95955c7e04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:56:10.831681 17930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c83e112172394012b09a659a2cf10a5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:56:10.835139 18014 tablet_bootstrap.cc:492] T c83e112172394012b09a659a2cf10a5d P 59ed1556867d4db3a023c7b887ad11cd: Bootstrap starting.
I20260929 15:56:10.837278 18015 tablet_bootstrap.cc:492] T c83e112172394012b09a659a2cf10a5d P cc75fb1ec51b40778f2d150548d45835: Bootstrap starting.
I20260929 15:56:10.839198 18014 tablet_bootstrap.cc:654] T c83e112172394012b09a659a2cf10a5d P 59ed1556867d4db3a023c7b887ad11cd: Neither blocks nor log segments found. Creating new log.
I20260929 15:56:10.840761 18014 log.cc:824] T c83e112172394012b09a659a2cf10a5d P 59ed1556867d4db3a023c7b887ad11cd: Log is configured to *not* fsync() on all Append() calls
I20260929 15:56:10.840885 18015 tablet_bootstrap.cc:654] T c83e112172394012b09a659a2cf10a5d P cc75fb1ec51b40778f2d150548d45835: Neither blocks nor log segments found. Creating new log.
I20260929 15:56:10.842088 18015 log.cc:824] T c83e112172394012b09a659a2cf10a5d P cc75fb1ec51b40778f2d150548d45835: Log is configured to *not* fsync() on all Append() calls
I20260929 15:56:10.844185 18014 tablet_bootstrap.cc:492] T c83e112172394012b09a659a2cf10a5d P 59ed1556867d4db3a023c7b887ad11cd: No bootstrap required, opened a new log
I20260929 15:56:10.844465 18014 ts_tablet_manager.cc:1397] T c83e112172394012b09a659a2cf10a5d P 59ed1556867d4db3a023c7b887ad11cd: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.002s
I20260929 15:56:10.845009 18015 tablet_bootstrap.cc:492] T c83e112172394012b09a659a2cf10a5d P cc75fb1ec51b40778f2d150548d45835: No bootstrap required, opened a new log
I20260929 15:56:10.845330 18015 ts_tablet_manager.cc:1397] T c83e112172394012b09a659a2cf10a5d P cc75fb1ec51b40778f2d150548d45835: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260929 15:56:10.845412 18018 tablet_bootstrap.cc:492] T c83e112172394012b09a659a2cf10a5d P 348ab908d1ac40faad1950ca9ad90571: Bootstrap starting.
I20260929 15:56:10.848330 18018 tablet_bootstrap.cc:654] T c83e112172394012b09a659a2cf10a5d P 348ab908d1ac40faad1950ca9ad90571: Neither blocks nor log segments found. Creating new log.
I20260929 15:56:10.849272 18018 log.cc:824] T c83e112172394012b09a659a2cf10a5d P 348ab908d1ac40faad1950ca9ad90571: Log is configured to *not* fsync() on all Append() calls
I20260929 15:56:10.852582 18018 tablet_bootstrap.cc:492] T c83e112172394012b09a659a2cf10a5d P 348ab908d1ac40faad1950ca9ad90571: No bootstrap required, opened a new log
I20260929 15:56:10.852859 18018 ts_tablet_manager.cc:1397] T c83e112172394012b09a659a2cf10a5d P 348ab908d1ac40faad1950ca9ad90571: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260929 15:56:10.855440 18014 raft_consensus.cc:348] T c83e112172394012b09a659a2cf10a5d P 59ed1556867d4db3a023c7b887ad11cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ed1556867d4db3a023c7b887ad11cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40389 } } peers { permanent_uuid: "cc75fb1ec51b40778f2d150548d45835" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 } } peers { permanent_uuid: "348ab908d1ac40faad1950ca9ad90571" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33799 } }
I20260929 15:56:10.855912 18014 raft_consensus.cc:374] T c83e112172394012b09a659a2cf10a5d P 59ed1556867d4db3a023c7b887ad11cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:56:10.856012 18014 raft_consensus.cc:729] T c83e112172394012b09a659a2cf10a5d P 59ed1556867d4db3a023c7b887ad11cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59ed1556867d4db3a023c7b887ad11cd, State: Initialized, Role: FOLLOWER
I20260929 15:56:10.855783 18015 raft_consensus.cc:348] T c83e112172394012b09a659a2cf10a5d P cc75fb1ec51b40778f2d150548d45835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ed1556867d4db3a023c7b887ad11cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40389 } } peers { permanent_uuid: "cc75fb1ec51b40778f2d150548d45835" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 } } peers { permanent_uuid: "348ab908d1ac40faad1950ca9ad90571" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33799 } }
I20260929 15:56:10.856189 18015 raft_consensus.cc:374] T c83e112172394012b09a659a2cf10a5d P cc75fb1ec51b40778f2d150548d45835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:56:10.856285 18015 raft_consensus.cc:729] T c83e112172394012b09a659a2cf10a5d P cc75fb1ec51b40778f2d150548d45835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc75fb1ec51b40778f2d150548d45835, State: Initialized, Role: FOLLOWER
I20260929 15:56:10.856742 18014 consensus_queue.cc:260] T c83e112172394012b09a659a2cf10a5d P 59ed1556867d4db3a023c7b887ad11cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ed1556867d4db3a023c7b887ad11cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40389 } } peers { permanent_uuid: "cc75fb1ec51b40778f2d150548d45835" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 } } peers { permanent_uuid: "348ab908d1ac40faad1950ca9ad90571" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33799 } }
I20260929 15:56:10.856825 18015 consensus_queue.cc:260] T c83e112172394012b09a659a2cf10a5d P cc75fb1ec51b40778f2d150548d45835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ed1556867d4db3a023c7b887ad11cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40389 } } peers { permanent_uuid: "cc75fb1ec51b40778f2d150548d45835" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 } } peers { permanent_uuid: "348ab908d1ac40faad1950ca9ad90571" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33799 } }
I20260929 15:56:10.858350 17862 heartbeater.cc:503] Master 127.0.97.62:35257 was elected leader, sending a full tablet report...
I20260929 15:56:10.858985 18014 ts_tablet_manager.cc:1428] T c83e112172394012b09a659a2cf10a5d P 59ed1556867d4db3a023c7b887ad11cd: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s
I20260929 15:56:10.861609 18015 ts_tablet_manager.cc:1428] T c83e112172394012b09a659a2cf10a5d P cc75fb1ec51b40778f2d150548d45835: Time spent starting tablet: real 0.016s user 0.006s sys 0.007s
I20260929 15:56:10.862314 18018 raft_consensus.cc:348] T c83e112172394012b09a659a2cf10a5d P 348ab908d1ac40faad1950ca9ad90571 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ed1556867d4db3a023c7b887ad11cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40389 } } peers { permanent_uuid: "cc75fb1ec51b40778f2d150548d45835" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 } } peers { permanent_uuid: "348ab908d1ac40faad1950ca9ad90571" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33799 } }
I20260929 15:56:10.862782 18018 raft_consensus.cc:374] T c83e112172394012b09a659a2cf10a5d P 348ab908d1ac40faad1950ca9ad90571 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:56:10.862931 18018 raft_consensus.cc:729] T c83e112172394012b09a659a2cf10a5d P 348ab908d1ac40faad1950ca9ad90571 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 348ab908d1ac40faad1950ca9ad90571, State: Initialized, Role: FOLLOWER
I20260929 15:56:10.863598 18018 consensus_queue.cc:260] T c83e112172394012b09a659a2cf10a5d P 348ab908d1ac40faad1950ca9ad90571 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ed1556867d4db3a023c7b887ad11cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40389 } } peers { permanent_uuid: "cc75fb1ec51b40778f2d150548d45835" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 } } peers { permanent_uuid: "348ab908d1ac40faad1950ca9ad90571" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33799 } }
I20260929 15:56:10.864769 17995 heartbeater.cc:503] Master 127.0.97.62:35257 was elected leader, sending a full tablet report...
I20260929 15:56:10.866195 18018 ts_tablet_manager.cc:1428] T c83e112172394012b09a659a2cf10a5d P 348ab908d1ac40faad1950ca9ad90571: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
I20260929 15:56:10.936033 18022 raft_consensus.cc:482] T c83e112172394012b09a659a2cf10a5d P 348ab908d1ac40faad1950ca9ad90571 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:56:10.936362 18022 raft_consensus.cc:504] T c83e112172394012b09a659a2cf10a5d P 348ab908d1ac40faad1950ca9ad90571 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ed1556867d4db3a023c7b887ad11cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40389 } } peers { permanent_uuid: "cc75fb1ec51b40778f2d150548d45835" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 } } peers { permanent_uuid: "348ab908d1ac40faad1950ca9ad90571" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33799 } }
I20260929 15:56:10.938060 18022 leader_election.cc:288] T c83e112172394012b09a659a2cf10a5d P 348ab908d1ac40faad1950ca9ad90571 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59ed1556867d4db3a023c7b887ad11cd (127.0.97.2:40389), cc75fb1ec51b40778f2d150548d45835 (127.0.97.1:41129)
W20260929 15:56:10.943746 17863 tablet.cc:2369] T c83e112172394012b09a659a2cf10a5d P 59ed1556867d4db3a023c7b887ad11cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:56:10.943996 17998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:10.938387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40389 (local address 127.0.97.3:57963)
0929 15:56:10.938730 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:10.938758 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:10.938941 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:10.939423 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:10.939429 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:10.939456 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:56:10.939631 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:10.939638 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:10.940625 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:10.940631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:10.942092 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:10.942104 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:10.943231 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:10.943237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:10.943425 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:10.943462 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 15:56:10.943558 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 15:56:10.943697 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":37}
I20260929 15:56:10.944063 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:10.938436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57963 (local address 127.0.97.2:40389)
0929 15:56:10.938655 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:10.938661 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:10.939086 (+ 425us) server_negotiation.cc:407] Connection header received
0929 15:56:10.939183 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:10.939188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:10.939274 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:10.939376 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:56:10.939778 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:10.940486 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:10.942239 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:10.942996 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:10.943191 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:10.943270 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:10.943601 (+ 331us) server_negotiation.cc:299] Negotiation successful
0929 15:56:10.943743 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 15:56:10.945055 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:10.938712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49651 (local address 127.0.97.1:41129)
0929 15:56:10.938886 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:10.938891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:10.939361 (+ 470us) server_negotiation.cc:407] Connection header received
0929 15:56:10.939603 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:10.939608 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:10.939667 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:10.939771 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:56:10.940248 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:10.941016 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:10.942782 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:10.943930 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:10.944054 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:10.944155 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:10.944663 (+ 508us) server_negotiation.cc:299] Negotiation successful
0929 15:56:10.944809 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 15:56:10.945065 17817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c83e112172394012b09a659a2cf10a5d" candidate_uuid: "348ab908d1ac40faad1950ca9ad90571" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59ed1556867d4db3a023c7b887ad11cd" is_pre_election: true
W20260929 15:56:10.945250 17729 tablet.cc:2369] T c83e112172394012b09a659a2cf10a5d P cc75fb1ec51b40778f2d150548d45835: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:56:10.945638 17817 raft_consensus.cc:2454] T c83e112172394012b09a659a2cf10a5d P 59ed1556867d4db3a023c7b887ad11cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 348ab908d1ac40faad1950ca9ad90571 in term 0.
I20260929 15:56:10.945839 18023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:10.938443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41129 (local address 127.0.97.3:49651)
0929 15:56:10.939254 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:10.939268 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:10.939446 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:10.939787 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:10.939791 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:10.939813 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:56:10.940102 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:10.940109 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:10.941160 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:10.941167 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:10.942639 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:10.942650 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:10.944291 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:10.944295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:10.944476 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:10.944505 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 15:56:10.945297 (+ 792us) client_negotiation.cc:240] Negotiation successful
0929 15:56:10.945456 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"mutex_wait_us":21,"thread_start_us":537,"threads_started":1}
I20260929 15:56:10.946327 17886 leader_election.cc:302] T c83e112172394012b09a659a2cf10a5d P 348ab908d1ac40faad1950ca9ad90571 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 348ab908d1ac40faad1950ca9ad90571, 59ed1556867d4db3a023c7b887ad11cd; no voters:
I20260929 15:56:10.946705 18022 raft_consensus.cc:2781] T c83e112172394012b09a659a2cf10a5d P 348ab908d1ac40faad1950ca9ad90571 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:56:10.946858 18022 raft_consensus.cc:482] T c83e112172394012b09a659a2cf10a5d P 348ab908d1ac40faad1950ca9ad90571 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:56:10.946971 18022 raft_consensus.cc:3037] T c83e112172394012b09a659a2cf10a5d P 348ab908d1ac40faad1950ca9ad90571 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:56:10.946846 17683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c83e112172394012b09a659a2cf10a5d" candidate_uuid: "348ab908d1ac40faad1950ca9ad90571" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc75fb1ec51b40778f2d150548d45835" is_pre_election: true
I20260929 15:56:10.947376 17683 raft_consensus.cc:2454] T c83e112172394012b09a659a2cf10a5d P cc75fb1ec51b40778f2d150548d45835 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 348ab908d1ac40faad1950ca9ad90571 in term 0.
I20260929 15:56:10.949653 18022 raft_consensus.cc:504] T c83e112172394012b09a659a2cf10a5d P 348ab908d1ac40faad1950ca9ad90571 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ed1556867d4db3a023c7b887ad11cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40389 } } peers { permanent_uuid: "cc75fb1ec51b40778f2d150548d45835" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 } } peers { permanent_uuid: "348ab908d1ac40faad1950ca9ad90571" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33799 } }
I20260929 15:56:10.950278 18022 leader_election.cc:288] T c83e112172394012b09a659a2cf10a5d P 348ab908d1ac40faad1950ca9ad90571 [CANDIDATE]: Term 1 election: Requested vote from peers 59ed1556867d4db3a023c7b887ad11cd (127.0.97.2:40389), cc75fb1ec51b40778f2d150548d45835 (127.0.97.1:41129)
I20260929 15:56:10.950654 17817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c83e112172394012b09a659a2cf10a5d" candidate_uuid: "348ab908d1ac40faad1950ca9ad90571" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59ed1556867d4db3a023c7b887ad11cd"
I20260929 15:56:10.950799 17683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c83e112172394012b09a659a2cf10a5d" candidate_uuid: "348ab908d1ac40faad1950ca9ad90571" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc75fb1ec51b40778f2d150548d45835"
I20260929 15:56:10.950860 17817 raft_consensus.cc:3037] T c83e112172394012b09a659a2cf10a5d P 59ed1556867d4db3a023c7b887ad11cd [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:56:10.950997 17683 raft_consensus.cc:3037] T c83e112172394012b09a659a2cf10a5d P cc75fb1ec51b40778f2d150548d45835 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:56:10.954070 17817 raft_consensus.cc:2454] T c83e112172394012b09a659a2cf10a5d P 59ed1556867d4db3a023c7b887ad11cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 348ab908d1ac40faad1950ca9ad90571 in term 1.
I20260929 15:56:10.954150 17683 raft_consensus.cc:2454] T c83e112172394012b09a659a2cf10a5d P cc75fb1ec51b40778f2d150548d45835 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 348ab908d1ac40faad1950ca9ad90571 in term 1.
I20260929 15:56:10.954474 17886 leader_election.cc:302] T c83e112172394012b09a659a2cf10a5d P 348ab908d1ac40faad1950ca9ad90571 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 348ab908d1ac40faad1950ca9ad90571, 59ed1556867d4db3a023c7b887ad11cd; no voters:
I20260929 15:56:10.954764 18022 raft_consensus.cc:2781] T c83e112172394012b09a659a2cf10a5d P 348ab908d1ac40faad1950ca9ad90571 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:56:10.955237 18022 raft_consensus.cc:686] T c83e112172394012b09a659a2cf10a5d P 348ab908d1ac40faad1950ca9ad90571 [term 1 LEADER]: Becoming Leader. State: Replica: 348ab908d1ac40faad1950ca9ad90571, State: Running, Role: LEADER
I20260929 15:56:10.955699 18022 consensus_queue.cc:237] T c83e112172394012b09a659a2cf10a5d P 348ab908d1ac40faad1950ca9ad90571 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ed1556867d4db3a023c7b887ad11cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40389 } } peers { permanent_uuid: "cc75fb1ec51b40778f2d150548d45835" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 } } peers { permanent_uuid: "348ab908d1ac40faad1950ca9ad90571" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33799 } }
I20260929 15:56:10.960318 17539 catalog_manager.cc:5495] T c83e112172394012b09a659a2cf10a5d P 348ab908d1ac40faad1950ca9ad90571 reported cstate change: term changed from 0 to 1, leader changed from <none> to 348ab908d1ac40faad1950ca9ad90571 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "348ab908d1ac40faad1950ca9ad90571" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ed1556867d4db3a023c7b887ad11cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc75fb1ec51b40778f2d150548d45835" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "348ab908d1ac40faad1950ca9ad90571" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33799 } health_report { overall_health: HEALTHY } } }
W20260929 15:56:10.969769 17996 tablet.cc:2369] T c83e112172394012b09a659a2cf10a5d P 348ab908d1ac40faad1950ca9ad90571: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:56:11.002691 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:10.997531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39276 (local address 127.0.97.3:33799)
0929 15:56:10.997712 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:10.997716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:10.997843 (+ 127us) server_negotiation.cc:407] Connection header received
0929 15:56:10.998028 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:10.998032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:10.998107 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:10.998229 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:56:10.998537 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:10.999252 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:11.000710 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:11.001103 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:11.001197 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:11.001663 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:56:11.002151 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:56:11.002263 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:11.002429 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 15:56:11.002493 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 15:56:11.011261 17817 raft_consensus.cc:1261] T c83e112172394012b09a659a2cf10a5d P 59ed1556867d4db3a023c7b887ad11cd [term 1 FOLLOWER]: Refusing update from remote peer 348ab908d1ac40faad1950ca9ad90571: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:56:11.011251 17683 raft_consensus.cc:1261] T c83e112172394012b09a659a2cf10a5d P cc75fb1ec51b40778f2d150548d45835 [term 1 FOLLOWER]: Refusing update from remote peer 348ab908d1ac40faad1950ca9ad90571: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:56:11.012455 18024 consensus_queue.cc:1035] T c83e112172394012b09a659a2cf10a5d P 348ab908d1ac40faad1950ca9ad90571 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ed1556867d4db3a023c7b887ad11cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:56:11.012845 18022 consensus_queue.cc:1035] T c83e112172394012b09a659a2cf10a5d P 348ab908d1ac40faad1950ca9ad90571 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc75fb1ec51b40778f2d150548d45835" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:56:11.026716 18028 mvcc.cc:205] Tried to move back new op lower bound from 7334696431650488320 to 7334696431438491648. Current Snapshot: MvccSnapshot[applied={T|T < 7334696431650488320}]
I20260929 15:56:11.040457 17537 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:56:11.040696 17537 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:56:11.044636 17537 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c74e657f349a4dccb341f46a38577015: Sending DeleteTablet for 3 replicas of tablet c83e112172394012b09a659a2cf10a5d
I20260929 15:56:11.045855 17797 tablet_service.cc:1514] Processing DeleteTablet for tablet c83e112172394012b09a659a2cf10a5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:56:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35706
I20260929 15:56:11.046248 17930 tablet_service.cc:1514] Processing DeleteTablet for tablet c83e112172394012b09a659a2cf10a5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:56:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39268
I20260929 15:56:11.046602 17663 tablet_service.cc:1514] Processing DeleteTablet for tablet c83e112172394012b09a659a2cf10a5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:56:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53572
I20260929 15:56:11.047199 18038 tablet_replica.cc:330] T c83e112172394012b09a659a2cf10a5d P 348ab908d1ac40faad1950ca9ad90571: stopping tablet replica
I20260929 15:56:11.047365 18037 tablet_replica.cc:330] T c83e112172394012b09a659a2cf10a5d P 59ed1556867d4db3a023c7b887ad11cd: stopping tablet replica
I20260929 15:56:11.047654 18038 raft_consensus.cc:2229] T c83e112172394012b09a659a2cf10a5d P 348ab908d1ac40faad1950ca9ad90571 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:56:11.047857 18037 raft_consensus.cc:2229] T c83e112172394012b09a659a2cf10a5d P 59ed1556867d4db3a023c7b887ad11cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:56:11.048419 18038 raft_consensus.cc:2258] T c83e112172394012b09a659a2cf10a5d P 348ab908d1ac40faad1950ca9ad90571 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:56:11.048509 18037 raft_consensus.cc:2258] T c83e112172394012b09a659a2cf10a5d P 59ed1556867d4db3a023c7b887ad11cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:56:11.049820 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 17600
I20260929 15:56:11.051144 18037 ts_tablet_manager.cc:1905] T c83e112172394012b09a659a2cf10a5d P 59ed1556867d4db3a023c7b887ad11cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:56:11.055234 18038 ts_tablet_manager.cc:1905] T c83e112172394012b09a659a2cf10a5d P 348ab908d1ac40faad1950ca9ad90571: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:56:11.059094 18037 ts_tablet_manager.cc:1918] T c83e112172394012b09a659a2cf10a5d P 59ed1556867d4db3a023c7b887ad11cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:56:11.059355 18037 log.cc:1196] T c83e112172394012b09a659a2cf10a5d P 59ed1556867d4db3a023c7b887ad11cd: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-1/wal/wals/c83e112172394012b09a659a2cf10a5d
I20260929 15:56:11.060030 18037 ts_tablet_manager.cc:1939] T c83e112172394012b09a659a2cf10a5d P 59ed1556867d4db3a023c7b887ad11cd: Deleting consensus metadata
I20260929 15:56:11.062243 18039 tablet_replica.cc:330] T c83e112172394012b09a659a2cf10a5d P cc75fb1ec51b40778f2d150548d45835: stopping tablet replica
I20260929 15:56:11.062178 17526 catalog_manager.cc:4841] TS 59ed1556867d4db3a023c7b887ad11cd (127.0.97.2:40389): tablet c83e112172394012b09a659a2cf10a5d (table test-table [id=9ac2006644f245bd8f76af95955c7e04]) successfully deleted
I20260929 15:56:11.063035 18039 raft_consensus.cc:2229] T c83e112172394012b09a659a2cf10a5d P cc75fb1ec51b40778f2d150548d45835 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:56:11.063670 18039 raft_consensus.cc:2258] T c83e112172394012b09a659a2cf10a5d P cc75fb1ec51b40778f2d150548d45835 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:56:11.065583 18038 ts_tablet_manager.cc:1918] T c83e112172394012b09a659a2cf10a5d P 348ab908d1ac40faad1950ca9ad90571: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:56:11.065760 18038 log.cc:1196] T c83e112172394012b09a659a2cf10a5d P 348ab908d1ac40faad1950ca9ad90571: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-2/wal/wals/c83e112172394012b09a659a2cf10a5d
I20260929 15:56:11.066200 18039 ts_tablet_manager.cc:1905] T c83e112172394012b09a659a2cf10a5d P cc75fb1ec51b40778f2d150548d45835: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:56:11.066303 18038 ts_tablet_manager.cc:1939] T c83e112172394012b09a659a2cf10a5d P 348ab908d1ac40faad1950ca9ad90571: Deleting consensus metadata
I20260929 15:56:11.067991 17526 catalog_manager.cc:4841] TS 348ab908d1ac40faad1950ca9ad90571 (127.0.97.3:33799): tablet c83e112172394012b09a659a2cf10a5d (table test-table [id=9ac2006644f245bd8f76af95955c7e04]) successfully deleted
Sep 29 15:56:11 dist-test-slave-r0ck krb5kdc[17494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 15:56:11.073535 18039 ts_tablet_manager.cc:1918] T c83e112172394012b09a659a2cf10a5d P cc75fb1ec51b40778f2d150548d45835: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:56:11.073707 18039 log.cc:1196] T c83e112172394012b09a659a2cf10a5d P cc75fb1ec51b40778f2d150548d45835: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697075657932-388-0/minicluster-data/ts-0/wal/wals/c83e112172394012b09a659a2cf10a5d
I20260929 15:56:11.074298 18039 ts_tablet_manager.cc:1939] T c83e112172394012b09a659a2cf10a5d P cc75fb1ec51b40778f2d150548d45835: Deleting consensus metadata
I20260929 15:56:11.076261 17523 catalog_manager.cc:4841] TS cc75fb1ec51b40778f2d150548d45835 (127.0.97.1:41129): tablet c83e112172394012b09a659a2cf10a5d (table test-table [id=9ac2006644f245bd8f76af95955c7e04]) successfully deleted
I20260929 15:56:11.079084 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:11.051673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53584 (local address 127.0.97.1:41129)
0929 15:56:11.051874 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:11.051883 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:11.062459 (+ 10576us) server_negotiation.cc:407] Connection header received
0929 15:56:11.062760 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:11.062765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:11.062876 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:11.062984 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:11.064922 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:11.066076 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:11.067973 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:11.068560 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:11.068636 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:11.072936 (+ 4300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:11.072950 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:11.072965 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:56:11.072997 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:11.076288 (+ 3291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:11.077039 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:11.077049 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:11.077054 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:11.077184 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:11.077603 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:11.077610 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:11.077613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:11.077949 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:11.078280 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:11.078505 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 15:56:11.078717 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 15:56:11.079758 17723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:56:11.222774 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 17600
I20260929 15:56:11.241092 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 17734
Sep 29 15:56:11 dist-test-slave-r0ck krb5kdc[17494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 15:56:11.260362 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:11.241985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35718 (local address 127.0.97.2:40389)
0929 15:56:11.242198 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:11.242205 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:11.242231 (+ 26us) server_negotiation.cc:407] Connection header received
0929 15:56:11.242418 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:11.242425 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:11.242536 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:11.242654 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:11.244335 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:11.245637 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:11.247376 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:11.248074 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:11.248264 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:11.252534 (+ 4270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:11.252553 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:11.252574 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:56:11.252629 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:11.256664 (+ 4035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:11.257728 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:11.257739 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:11.257744 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:11.257908 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:11.258453 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:11.258458 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:11.258460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:11.258980 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:11.259272 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:11.259679 (+ 407us) server_negotiation.cc:299] Negotiation successful
0929 15:56:11.259928 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 15:56:11.261101 17857 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:56:11.355477 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 17734
I20260929 15:56:11.380623 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 17867
Sep 29 15:56:11 dist-test-slave-r0ck krb5kdc[17494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 15:56:11.401924 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:11.381443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39284 (local address 127.0.97.3:33799)
0929 15:56:11.381625 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:11.381630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:11.381649 (+ 19us) server_negotiation.cc:407] Connection header received
0929 15:56:11.381862 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:11.381866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:11.381941 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:11.382025 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:11.384031 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:11.384860 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:11.386889 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:11.387677 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:11.387877 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:11.393913 (+ 6036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:11.393939 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:11.393962 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:56:11.394016 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:11.398501 (+ 4485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:11.399289 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:11.399307 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:11.399311 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:11.399519 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:11.399997 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:11.400005 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:11.400010 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:11.400517 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:11.400925 (+ 408us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:11.401123 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 15:56:11.401431 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 15:56:11.402701 17990 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:56:11.524761 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 17867
I20260929 15:56:11.544723 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 17507
I20260929 15:56:11.545853 17569 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:56:11.655638 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 17507
2026-09-29T15:56:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4357 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T15:56:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:56:11Z Disabled control of system clock
I20260929 15:56:11.731458 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45269
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 15:56:12.064746 18048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:56:12.065027 18048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:56:12.065109 18048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:56:12.072424 18048 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:56:12.072532 18048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:56:12.072569 18048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:56:12.072619 18048 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:56:12.072662 18048 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:56:12.081697 18048 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45269
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:56:12.082938 18048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:56:12.084591 18048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:56:12.096645 18056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:12.096632 18054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:12.097210 18053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:56:12.097590 18048 server_base.cc:1034] running on GCE node
I20260929 15:56:12.098409 18048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:56:12.099766 18048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:56:12.101006 18048 hybrid_clock.cc:648] HybridClock initialized: now 1790697372100969 us; error 45 us; skew 500 ppm
I20260929 15:56:12.101464 18048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:56:12.107620 18048 webserver.cc:460] Webserver started at http://127.0.97.62:46677/ using document root <none> and password file <none>
I20260929 15:56:12.108273 18048 fs_manager.cc:359] Metadata directory not provided
I20260929 15:56:12.108369 18048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:56:12.108634 18048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:56:12.111503 18048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "d1c225962315490fb1b0024bd232307d"
format_stamp: "Formatted at 2026-09-29 15:56:12 on dist-test-slave-r0ck"
server_key: "b135ef7cce930fbec1353ae8de7463ba"
server_key_iv: "db8650c56069df9c99f0883bb9b08c57"
server_key_version: "encryptionkey@0"
I20260929 15:56:12.112236 18048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "d1c225962315490fb1b0024bd232307d"
format_stamp: "Formatted at 2026-09-29 15:56:12 on dist-test-slave-r0ck"
server_key: "b135ef7cce930fbec1353ae8de7463ba"
server_key_iv: "db8650c56069df9c99f0883bb9b08c57"
server_key_version: "encryptionkey@0"
I20260929 15:56:12.117297 18048 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 15:56:12.120620 18062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:12.122103 18048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:56:12.122293 18048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "d1c225962315490fb1b0024bd232307d"
format_stamp: "Formatted at 2026-09-29 15:56:12 on dist-test-slave-r0ck"
server_key: "b135ef7cce930fbec1353ae8de7463ba"
server_key_iv: "db8650c56069df9c99f0883bb9b08c57"
server_key_version: "encryptionkey@0"
I20260929 15:56:12.122515 18048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:56:12.146381 18048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:56:12.147253 18048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:56:12.147552 18048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:56:12.165748 18048 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45269
I20260929 15:56:12.165768 18113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45269 every 8 connection(s)
I20260929 15:56:12.167112 18048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:56:12.170620 18114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:56:12.174106 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 18048
I20260929 15:56:12.174294 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:56:12.174853 388 external_mini_cluster.cc:1428] Setting key 9b1fc556e4b92594eb1f10c2f45e4990
I20260929 15:56:12.183071 18114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1c225962315490fb1b0024bd232307d: Bootstrap starting.
I20260929 15:56:12.186057 18114 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1c225962315490fb1b0024bd232307d: Neither blocks nor log segments found. Creating new log.
I20260929 15:56:12.187137 18114 log.cc:824] T 00000000000000000000000000000000 P d1c225962315490fb1b0024bd232307d: Log is configured to *not* fsync() on all Append() calls
I20260929 15:56:12.188171 18117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:12.176777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56030 (local address 127.0.97.62:45269)
0929 15:56:12.177713 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:12.177724 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:12.177764 (+ 40us) server_negotiation.cc:407] Connection header received
0929 15:56:12.179338 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:12.179357 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:12.180046 (+ 689us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:12.180508 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:12.181080 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:12.182381 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:12.183834 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:12.184518 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:12.184732 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:12.185898 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:12.185905 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:12.185917 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:12.186479 (+ 562us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:12.186570 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:12.186590 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:12.186715 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:12.186921 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 15:56:12.187201 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":313,"threads_started":1}
I20260929 15:56:12.189952 18114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1c225962315490fb1b0024bd232307d: No bootstrap required, opened a new log
I20260929 15:56:12.197077 18114 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1c225962315490fb1b0024bd232307d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c225962315490fb1b0024bd232307d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45269 } }
I20260929 15:56:12.197515 18114 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1c225962315490fb1b0024bd232307d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:56:12.197633 18114 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1c225962315490fb1b0024bd232307d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c225962315490fb1b0024bd232307d, State: Initialized, Role: FOLLOWER
I20260929 15:56:12.198139 18114 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1c225962315490fb1b0024bd232307d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c225962315490fb1b0024bd232307d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45269 } }
I20260929 15:56:12.198405 18114 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1c225962315490fb1b0024bd232307d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:56:12.198532 18114 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1c225962315490fb1b0024bd232307d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:56:12.198693 18114 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1c225962315490fb1b0024bd232307d [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:56:12.200855 18114 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1c225962315490fb1b0024bd232307d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c225962315490fb1b0024bd232307d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45269 } }
I20260929 15:56:12.201330 18114 leader_election.cc:302] T 00000000000000000000000000000000 P d1c225962315490fb1b0024bd232307d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1c225962315490fb1b0024bd232307d; no voters:
I20260929 15:56:12.202039 18114 leader_election.cc:288] T 00000000000000000000000000000000 P d1c225962315490fb1b0024bd232307d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:56:12.202201 18119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1c225962315490fb1b0024bd232307d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:56:12.203508 18119 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1c225962315490fb1b0024bd232307d [term 1 LEADER]: Becoming Leader. State: Replica: d1c225962315490fb1b0024bd232307d, State: Running, Role: LEADER
I20260929 15:56:12.204010 18119 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1c225962315490fb1b0024bd232307d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c225962315490fb1b0024bd232307d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45269 } }
I20260929 15:56:12.204167 18114 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1c225962315490fb1b0024bd232307d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:56:12.206501 18121 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1c225962315490fb1b0024bd232307d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1c225962315490fb1b0024bd232307d. Latest consensus state: current_term: 1 leader_uuid: "d1c225962315490fb1b0024bd232307d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c225962315490fb1b0024bd232307d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45269 } } }
I20260929 15:56:12.206872 18121 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1c225962315490fb1b0024bd232307d [sys.catalog]: This master's current role is: LEADER
I20260929 15:56:12.207410 18120 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1c225962315490fb1b0024bd232307d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1c225962315490fb1b0024bd232307d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c225962315490fb1b0024bd232307d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45269 } } }
I20260929 15:56:12.208060 18120 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1c225962315490fb1b0024bd232307d [sys.catalog]: This master's current role is: LEADER
I20260929 15:56:12.210426 18126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:56:12.215075 18126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:56:12.223367 18126 catalog_manager.cc:1347] Generated new cluster ID: a35988dcc5dd4d9fa944d08d3978aeec
I20260929 15:56:12.223484 18126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:56:12.233439 18126 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:56:12.235145 18126 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:56:12.245959 18126 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1c225962315490fb1b0024bd232307d: Generated new TSK 0
I20260929 15:56:12.246904 18126 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:56:12.256501 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45269
--builtin_ntp_servers=127.0.97.20:42105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 15:56:12.580407 18139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:56:12.580701 18139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:56:12.580874 18139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:56:12.588446 18139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:56:12.588583 18139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:56:12.588766 18139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:56:12.597719 18139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:56:12.598912 18139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:56:12.600659 18139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:56:12.613795 18145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:12.613884 18147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:12.613863 18144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:56:12.614464 18139 server_base.cc:1034] running on GCE node
I20260929 15:56:12.615005 18139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:56:12.616204 18139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:56:12.617527 18139 hybrid_clock.cc:648] HybridClock initialized: now 1790697372617478 us; error 36 us; skew 500 ppm
I20260929 15:56:12.617861 18139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:56:12.620100 18139 webserver.cc:460] Webserver started at http://127.0.97.1:33177/ using document root <none> and password file <none>
I20260929 15:56:12.620769 18139 fs_manager.cc:359] Metadata directory not provided
I20260929 15:56:12.620859 18139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:56:12.621214 18139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:56:12.623971 18139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-0/data/instance:
uuid: "74e5d9541b654fc88c189ae9a9a71554"
format_stamp: "Formatted at 2026-09-29 15:56:12 on dist-test-slave-r0ck"
server_key: "62aa4d68916dac497d6cbb815c1fb18c"
server_key_iv: "ab570ba242722b2b19a02b0c203c30eb"
server_key_version: "encryptionkey@0"
I20260929 15:56:12.624755 18139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "74e5d9541b654fc88c189ae9a9a71554"
format_stamp: "Formatted at 2026-09-29 15:56:12 on dist-test-slave-r0ck"
server_key: "62aa4d68916dac497d6cbb815c1fb18c"
server_key_iv: "ab570ba242722b2b19a02b0c203c30eb"
server_key_version: "encryptionkey@0"
I20260929 15:56:12.630002 18139 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 15:56:12.633299 18153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:12.634595 18139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 15:56:12.634796 18139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-0/wal
uuid: "74e5d9541b654fc88c189ae9a9a71554"
format_stamp: "Formatted at 2026-09-29 15:56:12 on dist-test-slave-r0ck"
server_key: "62aa4d68916dac497d6cbb815c1fb18c"
server_key_iv: "ab570ba242722b2b19a02b0c203c30eb"
server_key_version: "encryptionkey@0"
I20260929 15:56:12.635020 18139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:56:12.647003 18139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:56:12.647758 18139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:56:12.648032 18139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:56:12.648941 18139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:56:12.650408 18139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:56:12.650528 18139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:12.650681 18139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:56:12.650769 18139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:12.683133 18139 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:32797
I20260929 15:56:12.683178 18265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:32797 every 8 connection(s)
I20260929 15:56:12.684428 18139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:56:12.692210 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 18139
I20260929 15:56:12.692386 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:56:12.692807 388 external_mini_cluster.cc:1428] Setting key 48806742bb478663574691ab76359ba6
I20260929 15:56:12.695761 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45269
--builtin_ntp_servers=127.0.97.20:42105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 15:56:12.697532 18117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:12.687044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39277 (local address 127.0.97.62:45269)
0929 15:56:12.687230 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:12.687235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:12.688734 (+ 1499us) server_negotiation.cc:407] Connection header received
0929 15:56:12.690569 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:12.690575 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:12.690667 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:12.690791 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:12.691800 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:12.692623 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:12.694483 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:12.695080 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:12.695261 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:12.696564 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:12.696573 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:12.696581 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:12.696606 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:12.696649 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:12.696666 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:12.696763 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:12.697000 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 15:56:12.697112 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 15:56:12.698455 18268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:12.687541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45269 (local address 127.0.97.1:39277)
0929 15:56:12.688556 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:12.688612 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:12.690305 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:12.691011 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:12.691018 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:12.691099 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:56:12.691614 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:12.691629 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:12.692774 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:12.692780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:12.694320 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:12.694333 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:12.695343 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:12.695348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:12.695543 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:12.696243 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:56:12.696246 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:56:12.696294 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:56:12.696364 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:56:12.696765 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:56:12.696773 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:56:12.696779 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 15:56:12.696958 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 15:56:12.697221 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":331,"threads_started":1}
I20260929 15:56:12.699862 18266 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45269
I20260929 15:56:12.700121 18266 heartbeater.cc:464] Registering TS with master...
I20260929 15:56:12.700870 18266 heartbeater.cc:511] Master 127.0.97.62:45269 requested a full tablet report, sending...
I20260929 15:56:12.702893 18079 ts_manager.cc:194] Registered new tserver with Master: 74e5d9541b654fc88c189ae9a9a71554 (127.0.97.1:32797)
I20260929 15:56:12.704528 18079 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:39277
W20260929 15:56:13.021037 18269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:56:13.021334 18269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:56:13.021539 18269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:56:13.029219 18269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:56:13.029320 18269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:56:13.029527 18269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:56:13.038842 18269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:56:13.040164 18269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:56:13.041920 18269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:56:13.054862 18277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:13.054821 18275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:13.054903 18274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:56:13.058729 18269 server_base.cc:1034] running on GCE node
I20260929 15:56:13.059387 18269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:56:13.060454 18269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:56:13.061687 18269 hybrid_clock.cc:648] HybridClock initialized: now 1790697373061649 us; error 66 us; skew 500 ppm
I20260929 15:56:13.062026 18269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:56:13.064415 18269 webserver.cc:460] Webserver started at http://127.0.97.2:39683/ using document root <none> and password file <none>
I20260929 15:56:13.065135 18269 fs_manager.cc:359] Metadata directory not provided
I20260929 15:56:13.065304 18269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:56:13.065680 18269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:56:13.068604 18269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-1/data/instance:
uuid: "ab9f867985434c21a713f5f2dac0fef5"
format_stamp: "Formatted at 2026-09-29 15:56:13 on dist-test-slave-r0ck"
server_key: "2ca3096618e9d60daf0cf650ad45c070"
server_key_iv: "e695373963ab9dd8cf37ae7325be785b"
server_key_version: "encryptionkey@0"
I20260929 15:56:13.069475 18269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ab9f867985434c21a713f5f2dac0fef5"
format_stamp: "Formatted at 2026-09-29 15:56:13 on dist-test-slave-r0ck"
server_key: "2ca3096618e9d60daf0cf650ad45c070"
server_key_iv: "e695373963ab9dd8cf37ae7325be785b"
server_key_version: "encryptionkey@0"
I20260929 15:56:13.074747 18269 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 15:56:13.078418 18283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:13.079692 18269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 15:56:13.079861 18269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-1/wal
uuid: "ab9f867985434c21a713f5f2dac0fef5"
format_stamp: "Formatted at 2026-09-29 15:56:13 on dist-test-slave-r0ck"
server_key: "2ca3096618e9d60daf0cf650ad45c070"
server_key_iv: "e695373963ab9dd8cf37ae7325be785b"
server_key_version: "encryptionkey@0"
I20260929 15:56:13.080096 18269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:56:13.104096 18269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:56:13.105090 18269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:56:13.105458 18269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:56:13.106544 18269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:56:13.108079 18269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:56:13.108217 18269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:13.108354 18269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:56:13.108453 18269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:13.145089 18269 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44661
I20260929 15:56:13.145195 18395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44661 every 8 connection(s)
I20260929 15:56:13.146453 18269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:56:13.152004 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 18269
I20260929 15:56:13.152189 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:56:13.152736 388 external_mini_cluster.cc:1428] Setting key 0689234c32c3fc278526dc7a876fea5a
I20260929 15:56:13.155977 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45269
--builtin_ntp_servers=127.0.97.20:42105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 15:56:13.160151 18117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:13.149373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53347 (local address 127.0.97.62:45269)
0929 15:56:13.149616 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:13.149622 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:13.150796 (+ 1174us) server_negotiation.cc:407] Connection header received
0929 15:56:13.153180 (+ 2384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:13.153185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:13.153289 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:13.153477 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:13.154838 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:13.155568 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:13.157279 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:13.157758 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:13.157901 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:13.159334 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:13.159339 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:13.159342 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:13.159361 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:13.159393 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:13.159406 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:13.159505 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:13.159733 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 15:56:13.159834 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 15:56:13.161286 18398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:13.149748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45269 (local address 127.0.97.2:53347)
0929 15:56:13.150641 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:13.150681 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:13.152844 (+ 2163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:13.153697 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:13.153706 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:13.153829 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:56:13.154609 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:13.154630 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:13.155754 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:13.155767 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:13.157105 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:13.157117 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:13.157991 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:13.157998 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:13.158238 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:13.158961 (+ 723us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:56:13.158964 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:56:13.159004 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:56:13.159090 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:56:13.159502 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:56:13.159514 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:56:13.159524 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 15:56:13.159715 (+ 191us) client_negotiation.cc:240] Negotiation successful
0929 15:56:13.160118 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":273,"threads_started":1}
I20260929 15:56:13.162855 18396 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45269
I20260929 15:56:13.163167 18396 heartbeater.cc:464] Registering TS with master...
I20260929 15:56:13.164021 18396 heartbeater.cc:511] Master 127.0.97.62:45269 requested a full tablet report, sending...
I20260929 15:56:13.165685 18079 ts_manager.cc:194] Registered new tserver with Master: ab9f867985434c21a713f5f2dac0fef5 (127.0.97.2:44661)
I20260929 15:56:13.166993 18079 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:53347
W20260929 15:56:13.489703 18399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:56:13.489974 18399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:56:13.490059 18399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:56:13.497604 18399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:56:13.497727 18399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:56:13.497895 18399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:56:13.507138 18399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:56:13.508622 18399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:56:13.510936 18399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:56:13.525624 18407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:13.525653 18405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:13.525686 18404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:56:13.527401 18399 server_base.cc:1034] running on GCE node
I20260929 15:56:13.527997 18399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:56:13.528960 18399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:56:13.530194 18399 hybrid_clock.cc:648] HybridClock initialized: now 1790697373530141 us; error 30 us; skew 500 ppm
I20260929 15:56:13.530486 18399 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:56:13.532766 18399 webserver.cc:460] Webserver started at http://127.0.97.3:33379/ using document root <none> and password file <none>
I20260929 15:56:13.533519 18399 fs_manager.cc:359] Metadata directory not provided
I20260929 15:56:13.533620 18399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:56:13.533898 18399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:56:13.536743 18399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-2/data/instance:
uuid: "5708a7a76ae841fa8c60eb254a5091a0"
format_stamp: "Formatted at 2026-09-29 15:56:13 on dist-test-slave-r0ck"
server_key: "77e293fc36a5951c1b242a4a1a571fcc"
server_key_iv: "f548e0a685c68403cc928b2ffdea74aa"
server_key_version: "encryptionkey@0"
I20260929 15:56:13.537529 18399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "5708a7a76ae841fa8c60eb254a5091a0"
format_stamp: "Formatted at 2026-09-29 15:56:13 on dist-test-slave-r0ck"
server_key: "77e293fc36a5951c1b242a4a1a571fcc"
server_key_iv: "f548e0a685c68403cc928b2ffdea74aa"
server_key_version: "encryptionkey@0"
I20260929 15:56:13.543059 18399 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 15:56:13.546556 18413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:13.547900 18399 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 15:56:13.548082 18399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-2/wal
uuid: "5708a7a76ae841fa8c60eb254a5091a0"
format_stamp: "Formatted at 2026-09-29 15:56:13 on dist-test-slave-r0ck"
server_key: "77e293fc36a5951c1b242a4a1a571fcc"
server_key_iv: "f548e0a685c68403cc928b2ffdea74aa"
server_key_version: "encryptionkey@0"
I20260929 15:56:13.548352 18399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:56:13.584854 18399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:56:13.585812 18399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:56:13.586105 18399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:56:13.587250 18399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:56:13.588990 18399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:56:13.589218 18399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:13.589378 18399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:56:13.589488 18399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260929 15:56:13.624893 18399 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44437
I20260929 15:56:13.624989 18525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44437 every 8 connection(s)
I20260929 15:56:13.626598 18399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:56:13.632961 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 18399
I20260929 15:56:13.633174 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:56:13.633760 388 external_mini_cluster.cc:1428] Setting key 5dc8b9d61c8fbf36310e0060307d35e6
I20260929 15:56:13.640046 18117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:13.629287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34387 (local address 127.0.97.62:45269)
0929 15:56:13.629525 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:13.629531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:13.630899 (+ 1368us) server_negotiation.cc:407] Connection header received
0929 15:56:13.632785 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:13.632790 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:13.632863 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:13.632955 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:13.634429 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:13.635263 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:13.637111 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:13.637586 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:13.637686 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:13.639069 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:13.639073 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:13.639077 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:13.639094 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:13.639124 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:13.639135 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:13.639213 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:13.639593 (+ 380us) server_negotiation.cc:299] Negotiation successful
0929 15:56:13.639699 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260929 15:56:13.641074 18528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:13.629855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45269 (local address 127.0.97.3:34387)
0929 15:56:13.630749 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:13.630785 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:13.632422 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:13.633252 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:13.633264 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:13.633414 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:56:13.634223 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:13.634246 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:13.635503 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:13.635508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:13.636970 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:13.636982 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:13.637827 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:13.637832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:13.638043 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:13.638766 (+ 723us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:56:13.638768 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:56:13.638807 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:56:13.638890 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:56:13.639271 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:56:13.639280 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:56:13.639286 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 15:56:13.639574 (+ 288us) client_negotiation.cc:240] Negotiation successful
0929 15:56:13.639986 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":305,"threads_started":1}
I20260929 15:56:13.642488 18526 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45269
I20260929 15:56:13.642784 18526 heartbeater.cc:464] Registering TS with master...
I20260929 15:56:13.643507 18526 heartbeater.cc:511] Master 127.0.97.62:45269 requested a full tablet report, sending...
I20260929 15:56:13.644878 18079 ts_manager.cc:194] Registered new tserver with Master: 5708a7a76ae841fa8c60eb254a5091a0 (127.0.97.3:44437)
I20260929 15:56:13.645706 18079 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:34387
I20260929 15:56:13.649263 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:56:13.659543 18117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:13.653582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56044 (local address 127.0.97.62:45269)
0929 15:56:13.653817 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:13.653824 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:13.654118 (+ 294us) server_negotiation.cc:407] Connection header received
0929 15:56:13.654249 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:13.654253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:13.654324 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:13.654411 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:13.654854 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:13.655606 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:13.656995 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:13.657518 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:13.657623 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:13.658727 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:13.658732 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:13.658735 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:13.658752 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:13.658790 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:13.658803 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:13.658885 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:13.659002 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 15:56:13.659077 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 15:56:13.663880 18079 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:56:13.670595 18079 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:56:13.703487 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:13.684916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41374 (local address 127.0.97.1:32797)
0929 15:56:13.693016 (+ 8100us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:13.693028 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:13.694754 (+ 1726us) server_negotiation.cc:407] Connection header received
0929 15:56:13.695046 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:13.695053 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:13.695279 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:13.695443 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:56:13.696586 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:13.698396 (+ 1810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:13.700850 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:13.702283 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:13.702436 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:13.702698 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:13.702946 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 15:56:13.703121 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7875,"thread_start_us":7727,"threads_started":1}
I20260929 15:56:13.703502 18538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:13.683145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32797 (local address 127.0.0.1:41374)
0929 15:56:13.694631 (+ 11486us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:13.694650 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:13.694856 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:13.695765 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:13.695770 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:13.695796 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:56:13.696376 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:13.696391 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:13.698566 (+ 2175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:13.698572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:13.700624 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:13.700640 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:13.702487 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:13.702493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:13.702699 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:13.702740 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 15:56:13.702901 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 15:56:13.703103 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11338,"mutex_wait_us":45,"thread_start_us":9932,"threads_started":1}
I20260929 15:56:13.703717 18540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:13.683473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38780 (local address 127.0.97.2:44661)
0929 15:56:13.693630 (+ 10157us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:13.693642 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:13.693686 (+ 44us) server_negotiation.cc:407] Connection header received
0929 15:56:13.693996 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:13.694005 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:13.694269 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:13.694849 (+ 580us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:56:13.695921 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:13.697838 (+ 1917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:13.700813 (+ 2975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:13.701870 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:13.702025 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:13.702296 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:13.703035 (+ 739us) server_negotiation.cc:299] Negotiation successful
0929 15:56:13.703208 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9988,"thread_start_us":9857,"threads_started":1}
I20260929 15:56:13.704345 18539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:13.682887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44661 (local address 127.0.0.1:38780)
0929 15:56:13.693484 (+ 10597us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:13.693530 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:13.693705 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:13.695224 (+ 1519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:13.695236 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:13.695278 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:56:13.695716 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:13.695728 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:13.698057 (+ 2329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:13.698063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:13.700578 (+ 2515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:13.700603 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:13.702422 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:13.702429 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:13.702678 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:13.702741 (+ 63us) client_negotiation.cc:769] Sending connection context
0929 15:56:13.702889 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 15:56:13.703900 (+ 1011us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10330,"mutex_wait_us":842,"thread_start_us":11442,"threads_started":1}
I20260929 15:56:13.705163 18542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:13.694297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58884 (local address 127.0.97.3:44437)
0929 15:56:13.695932 (+ 1635us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:13.695940 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:13.696688 (+ 748us) server_negotiation.cc:407] Connection header received
0929 15:56:13.696901 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:13.696906 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:13.697045 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:13.697238 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:56:13.697761 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:13.699303 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:13.702007 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:13.703426 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:13.703574 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:13.703848 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:13.704611 (+ 763us) server_negotiation.cc:299] Negotiation successful
0929 15:56:13.704777 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1461,"thread_start_us":1287,"threads_started":1}
I20260929 15:56:13.705936 18543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:13.695904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44437 (local address 127.0.0.1:58884)
0929 15:56:13.696574 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:13.696599 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:13.696734 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:13.697214 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:13.697219 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:13.697244 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:56:13.697596 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:13.697605 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:13.699492 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:13.699497 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:13.701797 (+ 2300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:13.701814 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:13.704228 (+ 2414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:13.704242 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:13.704378 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:13.704429 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 15:56:13.705457 (+ 1028us) client_negotiation.cc:240] Negotiation successful
0929 15:56:13.705591 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":49,"thread_start_us":390,"threads_started":1}
I20260929 15:56:13.707964 18266 heartbeater.cc:503] Master 127.0.97.62:45269 was elected leader, sending a full tablet report...
I20260929 15:56:13.708043 18331 tablet_service.cc:1467] Processing CreateTablet for tablet 1a2a1e6362544d0e8a6292eefcf906bb (DEFAULT_TABLE table=test-table [id=e296512e7876481b848b1366f07d49a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:56:13.708087 18201 tablet_service.cc:1467] Processing CreateTablet for tablet 1a2a1e6362544d0e8a6292eefcf906bb (DEFAULT_TABLE table=test-table [id=e296512e7876481b848b1366f07d49a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:56:13.709532 18331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a2a1e6362544d0e8a6292eefcf906bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:56:13.709555 18201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a2a1e6362544d0e8a6292eefcf906bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:56:13.709275 18461 tablet_service.cc:1467] Processing CreateTablet for tablet 1a2a1e6362544d0e8a6292eefcf906bb (DEFAULT_TABLE table=test-table [id=e296512e7876481b848b1366f07d49a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:56:13.710613 18461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a2a1e6362544d0e8a6292eefcf906bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:56:13.720971 18544 tablet_bootstrap.cc:492] T 1a2a1e6362544d0e8a6292eefcf906bb P ab9f867985434c21a713f5f2dac0fef5: Bootstrap starting.
I20260929 15:56:13.723930 18545 tablet_bootstrap.cc:492] T 1a2a1e6362544d0e8a6292eefcf906bb P 74e5d9541b654fc88c189ae9a9a71554: Bootstrap starting.
I20260929 15:56:13.726794 18546 tablet_bootstrap.cc:492] T 1a2a1e6362544d0e8a6292eefcf906bb P 5708a7a76ae841fa8c60eb254a5091a0: Bootstrap starting.
I20260929 15:56:13.726972 18544 tablet_bootstrap.cc:654] T 1a2a1e6362544d0e8a6292eefcf906bb P ab9f867985434c21a713f5f2dac0fef5: Neither blocks nor log segments found. Creating new log.
I20260929 15:56:13.727965 18545 tablet_bootstrap.cc:654] T 1a2a1e6362544d0e8a6292eefcf906bb P 74e5d9541b654fc88c189ae9a9a71554: Neither blocks nor log segments found. Creating new log.
I20260929 15:56:13.728328 18544 log.cc:824] T 1a2a1e6362544d0e8a6292eefcf906bb P ab9f867985434c21a713f5f2dac0fef5: Log is configured to *not* fsync() on all Append() calls
I20260929 15:56:13.729189 18545 log.cc:824] T 1a2a1e6362544d0e8a6292eefcf906bb P 74e5d9541b654fc88c189ae9a9a71554: Log is configured to *not* fsync() on all Append() calls
I20260929 15:56:13.729719 18546 tablet_bootstrap.cc:654] T 1a2a1e6362544d0e8a6292eefcf906bb P 5708a7a76ae841fa8c60eb254a5091a0: Neither blocks nor log segments found. Creating new log.
I20260929 15:56:13.730602 18546 log.cc:824] T 1a2a1e6362544d0e8a6292eefcf906bb P 5708a7a76ae841fa8c60eb254a5091a0: Log is configured to *not* fsync() on all Append() calls
I20260929 15:56:13.731990 18544 tablet_bootstrap.cc:492] T 1a2a1e6362544d0e8a6292eefcf906bb P ab9f867985434c21a713f5f2dac0fef5: No bootstrap required, opened a new log
I20260929 15:56:13.732275 18544 ts_tablet_manager.cc:1397] T 1a2a1e6362544d0e8a6292eefcf906bb P ab9f867985434c21a713f5f2dac0fef5: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260929 15:56:13.733156 18546 tablet_bootstrap.cc:492] T 1a2a1e6362544d0e8a6292eefcf906bb P 5708a7a76ae841fa8c60eb254a5091a0: No bootstrap required, opened a new log
I20260929 15:56:13.733422 18546 ts_tablet_manager.cc:1397] T 1a2a1e6362544d0e8a6292eefcf906bb P 5708a7a76ae841fa8c60eb254a5091a0: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.005s
I20260929 15:56:13.740268 18545 tablet_bootstrap.cc:492] T 1a2a1e6362544d0e8a6292eefcf906bb P 74e5d9541b654fc88c189ae9a9a71554: No bootstrap required, opened a new log
I20260929 15:56:13.740579 18545 ts_tablet_manager.cc:1397] T 1a2a1e6362544d0e8a6292eefcf906bb P 74e5d9541b654fc88c189ae9a9a71554: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.007s
I20260929 15:56:13.741149 18546 raft_consensus.cc:348] T 1a2a1e6362544d0e8a6292eefcf906bb P 5708a7a76ae841fa8c60eb254a5091a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9f867985434c21a713f5f2dac0fef5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44661 } } peers { permanent_uuid: "74e5d9541b654fc88c189ae9a9a71554" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32797 } } peers { permanent_uuid: "5708a7a76ae841fa8c60eb254a5091a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44437 } }
I20260929 15:56:13.741621 18546 raft_consensus.cc:374] T 1a2a1e6362544d0e8a6292eefcf906bb P 5708a7a76ae841fa8c60eb254a5091a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:56:13.741724 18546 raft_consensus.cc:729] T 1a2a1e6362544d0e8a6292eefcf906bb P 5708a7a76ae841fa8c60eb254a5091a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5708a7a76ae841fa8c60eb254a5091a0, State: Initialized, Role: FOLLOWER
I20260929 15:56:13.742319 18546 consensus_queue.cc:260] T 1a2a1e6362544d0e8a6292eefcf906bb P 5708a7a76ae841fa8c60eb254a5091a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9f867985434c21a713f5f2dac0fef5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44661 } } peers { permanent_uuid: "74e5d9541b654fc88c189ae9a9a71554" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32797 } } peers { permanent_uuid: "5708a7a76ae841fa8c60eb254a5091a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44437 } }
I20260929 15:56:13.743438 18526 heartbeater.cc:503] Master 127.0.97.62:45269 was elected leader, sending a full tablet report...
I20260929 15:56:13.743945 18546 ts_tablet_manager.cc:1428] T 1a2a1e6362544d0e8a6292eefcf906bb P 5708a7a76ae841fa8c60eb254a5091a0: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260929 15:56:13.743916 18544 raft_consensus.cc:348] T 1a2a1e6362544d0e8a6292eefcf906bb P ab9f867985434c21a713f5f2dac0fef5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9f867985434c21a713f5f2dac0fef5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44661 } } peers { permanent_uuid: "74e5d9541b654fc88c189ae9a9a71554" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32797 } } peers { permanent_uuid: "5708a7a76ae841fa8c60eb254a5091a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44437 } }
I20260929 15:56:13.744329 18544 raft_consensus.cc:374] T 1a2a1e6362544d0e8a6292eefcf906bb P ab9f867985434c21a713f5f2dac0fef5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:56:13.744427 18544 raft_consensus.cc:729] T 1a2a1e6362544d0e8a6292eefcf906bb P ab9f867985434c21a713f5f2dac0fef5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab9f867985434c21a713f5f2dac0fef5, State: Initialized, Role: FOLLOWER
I20260929 15:56:13.745086 18544 consensus_queue.cc:260] T 1a2a1e6362544d0e8a6292eefcf906bb P ab9f867985434c21a713f5f2dac0fef5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9f867985434c21a713f5f2dac0fef5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44661 } } peers { permanent_uuid: "74e5d9541b654fc88c189ae9a9a71554" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32797 } } peers { permanent_uuid: "5708a7a76ae841fa8c60eb254a5091a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44437 } }
I20260929 15:56:13.750058 18396 heartbeater.cc:503] Master 127.0.97.62:45269 was elected leader, sending a full tablet report...
I20260929 15:56:13.750937 18544 ts_tablet_manager.cc:1428] T 1a2a1e6362544d0e8a6292eefcf906bb P ab9f867985434c21a713f5f2dac0fef5: Time spent starting tablet: real 0.019s user 0.012s sys 0.008s
I20260929 15:56:13.753978 18545 raft_consensus.cc:348] T 1a2a1e6362544d0e8a6292eefcf906bb P 74e5d9541b654fc88c189ae9a9a71554 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9f867985434c21a713f5f2dac0fef5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44661 } } peers { permanent_uuid: "74e5d9541b654fc88c189ae9a9a71554" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32797 } } peers { permanent_uuid: "5708a7a76ae841fa8c60eb254a5091a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44437 } }
I20260929 15:56:13.754354 18545 raft_consensus.cc:374] T 1a2a1e6362544d0e8a6292eefcf906bb P 74e5d9541b654fc88c189ae9a9a71554 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:56:13.754474 18545 raft_consensus.cc:729] T 1a2a1e6362544d0e8a6292eefcf906bb P 74e5d9541b654fc88c189ae9a9a71554 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74e5d9541b654fc88c189ae9a9a71554, State: Initialized, Role: FOLLOWER
I20260929 15:56:13.754761 18550 raft_consensus.cc:482] T 1a2a1e6362544d0e8a6292eefcf906bb P 5708a7a76ae841fa8c60eb254a5091a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:56:13.755010 18550 raft_consensus.cc:504] T 1a2a1e6362544d0e8a6292eefcf906bb P 5708a7a76ae841fa8c60eb254a5091a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9f867985434c21a713f5f2dac0fef5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44661 } } peers { permanent_uuid: "74e5d9541b654fc88c189ae9a9a71554" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32797 } } peers { permanent_uuid: "5708a7a76ae841fa8c60eb254a5091a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44437 } }
I20260929 15:56:13.755088 18545 consensus_queue.cc:260] T 1a2a1e6362544d0e8a6292eefcf906bb P 74e5d9541b654fc88c189ae9a9a71554 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9f867985434c21a713f5f2dac0fef5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44661 } } peers { permanent_uuid: "74e5d9541b654fc88c189ae9a9a71554" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32797 } } peers { permanent_uuid: "5708a7a76ae841fa8c60eb254a5091a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44437 } }
I20260929 15:56:13.756966 18550 leader_election.cc:288] T 1a2a1e6362544d0e8a6292eefcf906bb P 5708a7a76ae841fa8c60eb254a5091a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab9f867985434c21a713f5f2dac0fef5 (127.0.97.2:44661), 74e5d9541b654fc88c189ae9a9a71554 (127.0.97.1:32797)
I20260929 15:56:13.758068 18545 ts_tablet_manager.cc:1428] T 1a2a1e6362544d0e8a6292eefcf906bb P 74e5d9541b654fc88c189ae9a9a71554: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
I20260929 15:56:13.763808 18540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:13.757689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56041 (local address 127.0.97.2:44661)
0929 15:56:13.757882 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:13.757889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:13.757907 (+ 18us) server_negotiation.cc:407] Connection header received
0929 15:56:13.757967 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:13.757972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:13.758077 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:13.758189 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:56:13.758743 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:13.759922 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:13.762023 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:13.762802 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:13.762939 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:13.763041 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:13.763398 (+ 357us) server_negotiation.cc:299] Negotiation successful
0929 15:56:13.763547 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 15:56:13.763839 18528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:13.757166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44661 (local address 127.0.97.3:56041)
0929 15:56:13.757377 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:13.757399 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:13.757519 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:13.758248 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:13.758253 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:13.758288 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:56:13.758518 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:13.758537 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:13.760107 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:13.760113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:13.761860 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:13.761878 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:13.762962 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:13.762969 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:13.763221 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:13.763255 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 15:56:13.763357 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 15:56:13.763509 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":20}
I20260929 15:56:13.764612 18351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a2a1e6362544d0e8a6292eefcf906bb" candidate_uuid: "5708a7a76ae841fa8c60eb254a5091a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab9f867985434c21a713f5f2dac0fef5" is_pre_election: true
I20260929 15:56:13.765077 18351 raft_consensus.cc:2454] T 1a2a1e6362544d0e8a6292eefcf906bb P ab9f867985434c21a713f5f2dac0fef5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5708a7a76ae841fa8c60eb254a5091a0 in term 0.
I20260929 15:56:13.765853 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:13.757170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32797 (local address 127.0.97.3:44169)
0929 15:56:13.759737 (+ 2567us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:13.759765 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:13.760134 (+ 369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:13.760539 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:13.760543 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:13.760572 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:56:13.761017 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:13.761027 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:13.762069 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:13.762072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:13.763803 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:13.763818 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:13.765125 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:13.765129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:13.765297 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:13.765318 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 15:56:13.765443 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 15:56:13.765581 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2364,"mutex_wait_us":1142,"thread_start_us":330,"threads_started":1}
I20260929 15:56:13.765865 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:13.757530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44169 (local address 127.0.97.1:32797)
0929 15:56:13.757758 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:13.757764 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:13.759910 (+ 2146us) server_negotiation.cc:407] Connection header received
0929 15:56:13.760328 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:13.760337 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:13.760423 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:13.760523 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:56:13.761173 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:13.761913 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:13.764041 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:13.764844 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:13.764950 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:13.765018 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:13.765494 (+ 476us) server_negotiation.cc:299] Negotiation successful
0929 15:56:13.765629 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 15:56:13.765770 18417 leader_election.cc:302] T 1a2a1e6362544d0e8a6292eefcf906bb P 5708a7a76ae841fa8c60eb254a5091a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5708a7a76ae841fa8c60eb254a5091a0, ab9f867985434c21a713f5f2dac0fef5; no voters:
I20260929 15:56:13.766263 18550 raft_consensus.cc:2781] T 1a2a1e6362544d0e8a6292eefcf906bb P 5708a7a76ae841fa8c60eb254a5091a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:56:13.766431 18550 raft_consensus.cc:482] T 1a2a1e6362544d0e8a6292eefcf906bb P 5708a7a76ae841fa8c60eb254a5091a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:56:13.766553 18550 raft_consensus.cc:3037] T 1a2a1e6362544d0e8a6292eefcf906bb P 5708a7a76ae841fa8c60eb254a5091a0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:56:13.766794 18221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a2a1e6362544d0e8a6292eefcf906bb" candidate_uuid: "5708a7a76ae841fa8c60eb254a5091a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74e5d9541b654fc88c189ae9a9a71554" is_pre_election: true
I20260929 15:56:13.767316 18221 raft_consensus.cc:2454] T 1a2a1e6362544d0e8a6292eefcf906bb P 74e5d9541b654fc88c189ae9a9a71554 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5708a7a76ae841fa8c60eb254a5091a0 in term 0.
I20260929 15:56:13.769259 18550 raft_consensus.cc:504] T 1a2a1e6362544d0e8a6292eefcf906bb P 5708a7a76ae841fa8c60eb254a5091a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9f867985434c21a713f5f2dac0fef5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44661 } } peers { permanent_uuid: "74e5d9541b654fc88c189ae9a9a71554" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32797 } } peers { permanent_uuid: "5708a7a76ae841fa8c60eb254a5091a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44437 } }
I20260929 15:56:13.769929 18550 leader_election.cc:288] T 1a2a1e6362544d0e8a6292eefcf906bb P 5708a7a76ae841fa8c60eb254a5091a0 [CANDIDATE]: Term 1 election: Requested vote from peers ab9f867985434c21a713f5f2dac0fef5 (127.0.97.2:44661), 74e5d9541b654fc88c189ae9a9a71554 (127.0.97.1:32797)
I20260929 15:56:13.770355 18351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a2a1e6362544d0e8a6292eefcf906bb" candidate_uuid: "5708a7a76ae841fa8c60eb254a5091a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab9f867985434c21a713f5f2dac0fef5"
I20260929 15:56:13.770453 18221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a2a1e6362544d0e8a6292eefcf906bb" candidate_uuid: "5708a7a76ae841fa8c60eb254a5091a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74e5d9541b654fc88c189ae9a9a71554"
I20260929 15:56:13.770561 18351 raft_consensus.cc:3037] T 1a2a1e6362544d0e8a6292eefcf906bb P ab9f867985434c21a713f5f2dac0fef5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:56:13.770684 18221 raft_consensus.cc:3037] T 1a2a1e6362544d0e8a6292eefcf906bb P 74e5d9541b654fc88c189ae9a9a71554 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:56:13.772825 18351 raft_consensus.cc:2454] T 1a2a1e6362544d0e8a6292eefcf906bb P ab9f867985434c21a713f5f2dac0fef5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5708a7a76ae841fa8c60eb254a5091a0 in term 1.
I20260929 15:56:13.773203 18417 leader_election.cc:302] T 1a2a1e6362544d0e8a6292eefcf906bb P 5708a7a76ae841fa8c60eb254a5091a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5708a7a76ae841fa8c60eb254a5091a0, ab9f867985434c21a713f5f2dac0fef5; no voters:
I20260929 15:56:13.773551 18550 raft_consensus.cc:2781] T 1a2a1e6362544d0e8a6292eefcf906bb P 5708a7a76ae841fa8c60eb254a5091a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:56:13.773890 18221 raft_consensus.cc:2454] T 1a2a1e6362544d0e8a6292eefcf906bb P 74e5d9541b654fc88c189ae9a9a71554 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5708a7a76ae841fa8c60eb254a5091a0 in term 1.
I20260929 15:56:13.774047 18550 raft_consensus.cc:686] T 1a2a1e6362544d0e8a6292eefcf906bb P 5708a7a76ae841fa8c60eb254a5091a0 [term 1 LEADER]: Becoming Leader. State: Replica: 5708a7a76ae841fa8c60eb254a5091a0, State: Running, Role: LEADER
I20260929 15:56:13.774581 18550 consensus_queue.cc:237] T 1a2a1e6362544d0e8a6292eefcf906bb P 5708a7a76ae841fa8c60eb254a5091a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9f867985434c21a713f5f2dac0fef5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44661 } } peers { permanent_uuid: "74e5d9541b654fc88c189ae9a9a71554" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32797 } } peers { permanent_uuid: "5708a7a76ae841fa8c60eb254a5091a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44437 } }
I20260929 15:56:13.780522 18078 catalog_manager.cc:5495] T 1a2a1e6362544d0e8a6292eefcf906bb P 5708a7a76ae841fa8c60eb254a5091a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5708a7a76ae841fa8c60eb254a5091a0 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "5708a7a76ae841fa8c60eb254a5091a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9f867985434c21a713f5f2dac0fef5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74e5d9541b654fc88c189ae9a9a71554" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5708a7a76ae841fa8c60eb254a5091a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44437 } health_report { overall_health: HEALTHY } } }
I20260929 15:56:13.820709 18542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:13.815459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58898 (local address 127.0.97.3:44437)
0929 15:56:13.815645 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:13.815653 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:13.815677 (+ 24us) server_negotiation.cc:407] Connection header received
0929 15:56:13.815798 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:13.815802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:13.815886 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:13.816037 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:56:13.816423 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:13.817154 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:13.818590 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:13.819020 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:13.819156 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:13.819636 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:56:13.820130 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:56:13.820285 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:13.820382 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 15:56:13.820462 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 15:56:13.828758 18221 raft_consensus.cc:1261] T 1a2a1e6362544d0e8a6292eefcf906bb P 74e5d9541b654fc88c189ae9a9a71554 [term 1 FOLLOWER]: Refusing update from remote peer 5708a7a76ae841fa8c60eb254a5091a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:56:13.828787 18351 raft_consensus.cc:1261] T 1a2a1e6362544d0e8a6292eefcf906bb P ab9f867985434c21a713f5f2dac0fef5 [term 1 FOLLOWER]: Refusing update from remote peer 5708a7a76ae841fa8c60eb254a5091a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:56:13.829771 18550 consensus_queue.cc:1035] T 1a2a1e6362544d0e8a6292eefcf906bb P 5708a7a76ae841fa8c60eb254a5091a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74e5d9541b654fc88c189ae9a9a71554" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:56:13.830065 18554 consensus_queue.cc:1035] T 1a2a1e6362544d0e8a6292eefcf906bb P 5708a7a76ae841fa8c60eb254a5091a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab9f867985434c21a713f5f2dac0fef5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:56:13.841785 18557 mvcc.cc:205] Tried to move back new op lower bound from 7334696443193057280 to 7334696442986668032. Current Snapshot: MvccSnapshot[applied={T|T < 7334696443193057280}]
I20260929 15:56:13.859088 18078 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:56:13.859401 18078 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:56:13.863003 18078 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d1c225962315490fb1b0024bd232307d: Sending DeleteTablet for 3 replicas of tablet 1a2a1e6362544d0e8a6292eefcf906bb
I20260929 15:56:13.864084 18331 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a2a1e6362544d0e8a6292eefcf906bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:56:13 UTC) from {username='slave'} at 127.0.0.1:38780
I20260929 15:56:13.864302 18201 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a2a1e6362544d0e8a6292eefcf906bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:56:13 UTC) from {username='slave'} at 127.0.0.1:41374
I20260929 15:56:13.864485 18461 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a2a1e6362544d0e8a6292eefcf906bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:56:13 UTC) from {username='slave'} at 127.0.0.1:58884
I20260929 15:56:13.866757 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 18139
I20260929 15:56:13.876295 18568 tablet_replica.cc:330] T 1a2a1e6362544d0e8a6292eefcf906bb P 74e5d9541b654fc88c189ae9a9a71554: stopping tablet replica
I20260929 15:56:13.876899 18568 raft_consensus.cc:2229] T 1a2a1e6362544d0e8a6292eefcf906bb P 74e5d9541b654fc88c189ae9a9a71554 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:56:13.877786 18568 raft_consensus.cc:2258] T 1a2a1e6362544d0e8a6292eefcf906bb P 74e5d9541b654fc88c189ae9a9a71554 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:56:13.878940 18569 tablet_replica.cc:330] T 1a2a1e6362544d0e8a6292eefcf906bb P 5708a7a76ae841fa8c60eb254a5091a0: stopping tablet replica
I20260929 15:56:13.879452 18569 raft_consensus.cc:2229] T 1a2a1e6362544d0e8a6292eefcf906bb P 5708a7a76ae841fa8c60eb254a5091a0 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:56:13.880290 18569 raft_consensus.cc:2258] T 1a2a1e6362544d0e8a6292eefcf906bb P 5708a7a76ae841fa8c60eb254a5091a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:56:13.882643 18568 ts_tablet_manager.cc:1905] T 1a2a1e6362544d0e8a6292eefcf906bb P 74e5d9541b654fc88c189ae9a9a71554: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:56:13.882936 18567 tablet_replica.cc:330] T 1a2a1e6362544d0e8a6292eefcf906bb P ab9f867985434c21a713f5f2dac0fef5: stopping tablet replica
I20260929 15:56:13.884619 18569 ts_tablet_manager.cc:1905] T 1a2a1e6362544d0e8a6292eefcf906bb P 5708a7a76ae841fa8c60eb254a5091a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:56:13.887562 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:13.868398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41382 (local address 127.0.97.1:32797)
0929 15:56:13.868631 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:13.868640 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:13.876146 (+ 7506us) server_negotiation.cc:407] Connection header received
0929 15:56:13.876389 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:13.876396 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:13.876501 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:13.876612 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:13.878122 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:13.879302 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:13.882411 (+ 3109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:13.883061 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:13.883742 (+ 681us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:13.885454 (+ 1712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:13.885466 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:13.885482 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:13.886238 (+ 756us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:13.886300 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:13.886325 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:13.886440 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:13.887062 (+ 622us) server_negotiation.cc:299] Negotiation successful
0929 15:56:13.887184 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 15:56:13.888656 18261 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:56:13.889482 18567 raft_consensus.cc:2229] T 1a2a1e6362544d0e8a6292eefcf906bb P ab9f867985434c21a713f5f2dac0fef5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:56:13.890112 18567 raft_consensus.cc:2258] T 1a2a1e6362544d0e8a6292eefcf906bb P ab9f867985434c21a713f5f2dac0fef5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:56:13.892365 18567 ts_tablet_manager.cc:1905] T 1a2a1e6362544d0e8a6292eefcf906bb P ab9f867985434c21a713f5f2dac0fef5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:56:13.893527 18569 ts_tablet_manager.cc:1918] T 1a2a1e6362544d0e8a6292eefcf906bb P 5708a7a76ae841fa8c60eb254a5091a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:56:13.893702 18569 log.cc:1196] T 1a2a1e6362544d0e8a6292eefcf906bb P 5708a7a76ae841fa8c60eb254a5091a0: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-2/wal/wals/1a2a1e6362544d0e8a6292eefcf906bb
I20260929 15:56:13.894279 18569 ts_tablet_manager.cc:1939] T 1a2a1e6362544d0e8a6292eefcf906bb P 5708a7a76ae841fa8c60eb254a5091a0: Deleting consensus metadata
I20260929 15:56:13.896165 18063 catalog_manager.cc:4841] TS 5708a7a76ae841fa8c60eb254a5091a0 (127.0.97.3:44437): tablet 1a2a1e6362544d0e8a6292eefcf906bb (table test-table [id=e296512e7876481b848b1366f07d49a6]) successfully deleted
I20260929 15:56:13.900717 18567 ts_tablet_manager.cc:1918] T 1a2a1e6362544d0e8a6292eefcf906bb P ab9f867985434c21a713f5f2dac0fef5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:56:13.900970 18567 log.cc:1196] T 1a2a1e6362544d0e8a6292eefcf906bb P ab9f867985434c21a713f5f2dac0fef5: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697075657932-388-0/minicluster-data/ts-1/wal/wals/1a2a1e6362544d0e8a6292eefcf906bb
I20260929 15:56:13.901614 18567 ts_tablet_manager.cc:1939] T 1a2a1e6362544d0e8a6292eefcf906bb P ab9f867985434c21a713f5f2dac0fef5: Deleting consensus metadata
I20260929 15:56:13.903439 18066 catalog_manager.cc:4841] TS ab9f867985434c21a713f5f2dac0fef5 (127.0.97.2:44661): tablet 1a2a1e6362544d0e8a6292eefcf906bb (table test-table [id=e296512e7876481b848b1366f07d49a6]) successfully deleted
I20260929 15:56:13.988631 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 18139
I20260929 15:56:14.013885 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 18269
W20260929 15:56:14.013973 18063 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:32797 (error 108)
I20260929 15:56:14.016912 18539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:14.016388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32797 (local address 127.0.0.1:41390)
0929 15:56:14.016593 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:14.016718 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260929 15:56:14.017306 18063 catalog_manager.cc:4584] TS 74e5d9541b654fc88c189ae9a9a71554 (127.0.97.1:32797): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1a2a1e6362544d0e8a6292eefcf906bb: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32797: connect: Connection refused (error 111)
I20260929 15:56:14.022281 18540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:14.014826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38782 (local address 127.0.97.2:44661)
0929 15:56:14.015027 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:14.015034 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:14.015171 (+ 137us) server_negotiation.cc:407] Connection header received
0929 15:56:14.015395 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:14.015401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:14.015528 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:14.015665 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:14.016283 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:14.017205 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:14.019242 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:14.019664 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:14.019764 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:14.020929 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:14.020937 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:14.020952 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:14.021633 (+ 681us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:14.021679 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:14.021699 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:14.021785 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:14.021950 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 15:56:14.022039 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 15:56:14.023059 18391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:56:14.071296 18539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:14.070721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32797 (local address 127.0.0.1:41398)
0929 15:56:14.070982 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:14.071094 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260929 15:56:14.072501 18539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:14.072076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32797 (local address 127.0.0.1:41410)
0929 15:56:14.072247 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:14.072349 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 15:56:14.113225 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 18269
I20260929 15:56:14.130649 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 18399
I20260929 15:56:14.133271 18539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:14.132736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32797 (local address 127.0.0.1:41418)
0929 15:56:14.132957 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:14.133063 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 15:56:14.134461 18539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:14.134079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32797 (local address 127.0.0.1:41434)
0929 15:56:14.134238 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:14.134330 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 15:56:14.139972 18542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:14.131599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58908 (local address 127.0.97.3:44437)
0929 15:56:14.131800 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:14.131804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:14.131825 (+ 21us) server_negotiation.cc:407] Connection header received
0929 15:56:14.132350 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:14.132356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:14.132452 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:14.132593 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:14.133082 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:14.134230 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:14.136274 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:14.136741 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:14.136853 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:14.138289 (+ 1436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:14.138301 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:14.138315 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:14.138950 (+ 635us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:14.139001 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:14.139024 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:14.139108 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:14.139287 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 15:56:14.139368 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 15:56:14.140719 18521 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:56:14.214872 18539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:14.214412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32797 (local address 127.0.0.1:41438)
0929 15:56:14.214625 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:14.214721 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 15:56:14.215932 18539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:14.215559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32797 (local address 127.0.0.1:41442)
0929 15:56:14.215762 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:14.215827 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260929 15:56:14.263132 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 18399
I20260929 15:56:14.282629 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 18048
I20260929 15:56:14.283739 18109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:56:14.393939 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 18048
I20260929 15:56:14.393980 18539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:14.393303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32797 (local address 127.0.0.1:41446)
0929 15:56:14.393648 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:14.393768 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194}
2026-09-29T15:56:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2740 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:56:14 dist-test-slave-r0ck krb5kdc[18578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:56:14 dist-test-slave-r0ck krb5kdc[18578](info): set up 2 sockets
Sep 29 15:56:14 dist-test-slave-r0ck krb5kdc[18578](info): commencing operation
krb5kdc: starting...
W20260929 15:56:15.517292 388 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:56:15 dist-test-slave-r0ck krb5kdc[18578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:56:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:56:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:56:15.770816 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33569
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:56:16.118191 18591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:56:16.118548 18591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:56:16.118716 18591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:56:16.126037 18591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:56:16.126183 18591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:56:16.126298 18591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:56:16.126408 18591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:56:16.126485 18591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:56:16.136529 18591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:56:16.137938 18591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:56:16.139286 18591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:56:16.151366 18597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:16.151405 18599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:16.151443 18596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:56:16.152424 18591 server_base.cc:1034] running on GCE node
I20260929 15:56:16.153108 18591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:56:16.154487 18591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:56:16.155738 18591 hybrid_clock.cc:648] HybridClock initialized: now 1790697376155653 us; error 47 us; skew 500 ppm
I20260929 15:56:16.156122 18591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:56:16 dist-test-slave-r0ck krb5kdc[18578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:56:16.159334 18591 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:56:16.160851 18591 webserver.cc:460] Webserver started at http://127.0.97.62:34505/ using document root <none> and password file <none>
I20260929 15:56:16.161590 18591 fs_manager.cc:359] Metadata directory not provided
I20260929 15:56:16.161736 18591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:56:16.162056 18591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:56:16.164970 18591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "0657ba45a321475bb0006b040ac5fc04"
format_stamp: "Formatted at 2026-09-29 15:56:16 on dist-test-slave-r0ck"
server_key: "8c5c99648ded3beef4ef488c2d38226a"
server_key_iv: "9fb6b58503d82506a0ada221126ad753"
server_key_version: "encryptionkey@0"
I20260929 15:56:16.165758 18591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "0657ba45a321475bb0006b040ac5fc04"
format_stamp: "Formatted at 2026-09-29 15:56:16 on dist-test-slave-r0ck"
server_key: "8c5c99648ded3beef4ef488c2d38226a"
server_key_iv: "9fb6b58503d82506a0ada221126ad753"
server_key_version: "encryptionkey@0"
I20260929 15:56:16.170877 18591 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 15:56:16.174299 18606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:16.175612 18591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:56:16.175801 18591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "0657ba45a321475bb0006b040ac5fc04"
format_stamp: "Formatted at 2026-09-29 15:56:16 on dist-test-slave-r0ck"
server_key: "8c5c99648ded3beef4ef488c2d38226a"
server_key_iv: "9fb6b58503d82506a0ada221126ad753"
server_key_version: "encryptionkey@0"
I20260929 15:56:16.175999 18591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:56:16.196704 18591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:56:16.201143 18591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:56:16.201463 18591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:56:16.221901 18591 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33569
I20260929 15:56:16.221931 18657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33569 every 8 connection(s)
I20260929 15:56:16.223259 18591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:56:16.225816 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 18591
I20260929 15:56:16.226027 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:56:16.226490 388 external_mini_cluster.cc:1428] Setting key a676b34ea7c711c4dec562a607120840
I20260929 15:56:16.227210 18658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 15:56:16 dist-test-slave-r0ck krb5kdc[18578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:56:16.243969 18658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0657ba45a321475bb0006b040ac5fc04: Bootstrap starting.
I20260929 15:56:16.248201 18658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0657ba45a321475bb0006b040ac5fc04: Neither blocks nor log segments found. Creating new log.
I20260929 15:56:16.249456 18658 log.cc:824] T 00000000000000000000000000000000 P 0657ba45a321475bb0006b040ac5fc04: Log is configured to *not* fsync() on all Append() calls
I20260929 15:56:16.253054 18661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:16.228350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55000 (local address 127.0.97.62:33569)
0929 15:56:16.229407 (+ 1057us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:16.229426 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:16.229494 (+ 68us) server_negotiation.cc:407] Connection header received
0929 15:56:16.232078 (+ 2584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:16.232110 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:16.232649 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:16.233098 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:16.234556 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:16.236889 (+ 2333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:16.238400 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:16.239746 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:16.239943 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:16.243981 (+ 4038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:16.243994 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:16.244014 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:56:16.244063 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:16.247963 (+ 3900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:16.248730 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:16.248738 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:16.248746 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:16.248940 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:16.249328 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:16.249333 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:16.249336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:16.250363 (+ 1027us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:16.250811 (+ 448us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:16.251353 (+ 542us) server_negotiation.cc:299] Negotiation successful
0929 15:56:16.251785 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":723,"thread_start_us":441,"threads_started":1}
I20260929 15:56:16.253103 18658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0657ba45a321475bb0006b040ac5fc04: No bootstrap required, opened a new log
I20260929 15:56:16.261083 18658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0657ba45a321475bb0006b040ac5fc04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0657ba45a321475bb0006b040ac5fc04" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33569 } }
I20260929 15:56:16.261485 18658 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0657ba45a321475bb0006b040ac5fc04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:56:16.261592 18658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0657ba45a321475bb0006b040ac5fc04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0657ba45a321475bb0006b040ac5fc04, State: Initialized, Role: FOLLOWER
I20260929 15:56:16.262265 18658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0657ba45a321475bb0006b040ac5fc04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0657ba45a321475bb0006b040ac5fc04" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33569 } }
I20260929 15:56:16.262532 18658 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0657ba45a321475bb0006b040ac5fc04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:56:16.262703 18658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0657ba45a321475bb0006b040ac5fc04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:56:16.262874 18658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0657ba45a321475bb0006b040ac5fc04 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:56:16.265031 18658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0657ba45a321475bb0006b040ac5fc04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0657ba45a321475bb0006b040ac5fc04" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33569 } }
I20260929 15:56:16.265630 18658 leader_election.cc:302] T 00000000000000000000000000000000 P 0657ba45a321475bb0006b040ac5fc04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0657ba45a321475bb0006b040ac5fc04; no voters:
I20260929 15:56:16.266360 18658 leader_election.cc:288] T 00000000000000000000000000000000 P 0657ba45a321475bb0006b040ac5fc04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:56:16.266485 18663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0657ba45a321475bb0006b040ac5fc04 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:56:16.267256 18663 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0657ba45a321475bb0006b040ac5fc04 [term 1 LEADER]: Becoming Leader. State: Replica: 0657ba45a321475bb0006b040ac5fc04, State: Running, Role: LEADER
I20260929 15:56:16.267803 18663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0657ba45a321475bb0006b040ac5fc04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0657ba45a321475bb0006b040ac5fc04" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33569 } }
I20260929 15:56:16.268550 18658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0657ba45a321475bb0006b040ac5fc04 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:56:16.270673 18664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0657ba45a321475bb0006b040ac5fc04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0657ba45a321475bb0006b040ac5fc04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0657ba45a321475bb0006b040ac5fc04" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33569 } } }
I20260929 15:56:16.270684 18665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0657ba45a321475bb0006b040ac5fc04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0657ba45a321475bb0006b040ac5fc04. Latest consensus state: current_term: 1 leader_uuid: "0657ba45a321475bb0006b040ac5fc04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0657ba45a321475bb0006b040ac5fc04" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33569 } } }
I20260929 15:56:16.271135 18664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0657ba45a321475bb0006b040ac5fc04 [sys.catalog]: This master's current role is: LEADER
I20260929 15:56:16.271135 18665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0657ba45a321475bb0006b040ac5fc04 [sys.catalog]: This master's current role is: LEADER
I20260929 15:56:16.274537 18669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:56:16.279227 18669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:56:16.288476 18669 catalog_manager.cc:1347] Generated new cluster ID: f76dd892bf67456ea94f91898cf4a5c4
I20260929 15:56:16.288648 18669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:56:16.304054 18669 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:56:16.305447 18669 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:56:16.324568 18669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0657ba45a321475bb0006b040ac5fc04: Generated new TSK 0
I20260929 15:56:16.325820 18669 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 15:56:16.393172 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33569
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:56:16.733705 18684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:56:16.734093 18684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:56:16.734264 18684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:56:16.742641 18684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:56:16.742765 18684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:56:16.742887 18684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:56:16.753635 18684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33569
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:56:16.755076 18684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:56:16.756409 18684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:56:16.769613 18690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:16.769744 18692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:16.769616 18689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:56:16.770885 18684 server_base.cc:1034] running on GCE node
I20260929 15:56:16.771783 18684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:56:16.773136 18684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:56:16.774421 18684 hybrid_clock.cc:648] HybridClock initialized: now 1790697376774320 us; error 115 us; skew 500 ppm
I20260929 15:56:16.774842 18684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:56:16 dist-test-slave-r0ck krb5kdc[18578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:56:16.778110 18684 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:56:16.779623 18684 webserver.cc:460] Webserver started at http://127.0.97.1:33421/ using document root <none> and password file <none>
I20260929 15:56:16.780311 18684 fs_manager.cc:359] Metadata directory not provided
I20260929 15:56:16.780454 18684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:56:16.780805 18684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:56:16.783957 18684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-0/data/instance:
uuid: "12070c00f03f468b8ad0e99a6a36b073"
format_stamp: "Formatted at 2026-09-29 15:56:16 on dist-test-slave-r0ck"
server_key: "abd781857892eb3151cb9cf730278a65"
server_key_iv: "e89988dfa13ea1c964ff5d574961aa5f"
server_key_version: "encryptionkey@0"
I20260929 15:56:16.784793 18684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "12070c00f03f468b8ad0e99a6a36b073"
format_stamp: "Formatted at 2026-09-29 15:56:16 on dist-test-slave-r0ck"
server_key: "abd781857892eb3151cb9cf730278a65"
server_key_iv: "e89988dfa13ea1c964ff5d574961aa5f"
server_key_version: "encryptionkey@0"
I20260929 15:56:16.789891 18684 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260929 15:56:16.794227 18699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:16.795625 18684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:56:16.795806 18684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-0/wal
uuid: "12070c00f03f468b8ad0e99a6a36b073"
format_stamp: "Formatted at 2026-09-29 15:56:16 on dist-test-slave-r0ck"
server_key: "abd781857892eb3151cb9cf730278a65"
server_key_iv: "e89988dfa13ea1c964ff5d574961aa5f"
server_key_version: "encryptionkey@0"
I20260929 15:56:16.796022 18684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:56:16.820498 18684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:56:16.825155 18684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:56:16.825502 18684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:56:16.826469 18684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:56:16.827915 18684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:56:16.828022 18684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:16.828176 18684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:56:16.828248 18684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:16.862599 18684 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35355
I20260929 15:56:16.862686 18811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35355 every 8 connection(s)
I20260929 15:56:16.863840 18684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:56:16.869931 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 18684
I20260929 15:56:16.870116 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:56:16.870671 388 external_mini_cluster.cc:1428] Setting key 81fdabaf52b8c11b7be1b6dd1a0da04f
Sep 29 15:56:16 dist-test-slave-r0ck krb5kdc[18578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:56:16.887964 18814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:16.866410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38327 (local address 127.0.97.62:33569)
0929 15:56:16.866877 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:16.866881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:16.867640 (+ 759us) server_negotiation.cc:407] Connection header received
0929 15:56:16.869667 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:16.869671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:16.869738 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:16.869900 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:16.871848 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:16.872938 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:16.876283 (+ 3345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:16.876804 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:16.876975 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:16.881910 (+ 4935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:16.881916 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:16.881919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:56:16.881962 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:16.884695 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:16.885590 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:16.885600 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:16.885603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:16.885720 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:16.886161 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:16.886166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:16.886170 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:16.886458 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:16.886635 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:16.887347 (+ 712us) server_negotiation.cc:299] Negotiation successful
0929 15:56:16.887586 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":253,"threads_started":1}
I20260929 15:56:16.888967 18815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:16.866787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33569 (local address 127.0.97.1:38327)
0929 15:56:16.867485 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:16.867529 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:16.869326 (+ 1797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:16.870117 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:16.870130 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:16.870862 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:56:16.871602 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:16.871627 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:16.873227 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:16.873236 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:16.876027 (+ 2791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:16.876052 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:16.877121 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:16.877131 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:16.877516 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:16.878426 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:56:16.878461 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:56:16.881609 (+ 3148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:56:16.884906 (+ 3297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:56:16.884917 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:56:16.884937 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:56:16.885448 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:56:16.885830 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:56:16.885837 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:56:16.885843 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:56:16.886035 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:56:16.886589 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:56:16.886597 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:56:16.886956 (+ 359us) client_negotiation.cc:769] Sending connection context
0929 15:56:16.887229 (+ 273us) client_negotiation.cc:240] Negotiation successful
0929 15:56:16.887784 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":279,"threads_started":1}
I20260929 15:56:16.890777 18812 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33569
I20260929 15:56:16.891235 18812 heartbeater.cc:464] Registering TS with master...
I20260929 15:56:16.892392 18812 heartbeater.cc:511] Master 127.0.97.62:33569 requested a full tablet report, sending...
I20260929 15:56:16.895401 18623 ts_manager.cc:194] Registered new tserver with Master: 12070c00f03f468b8ad0e99a6a36b073 (127.0.97.1:35355)
I20260929 15:56:16.897941 18623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:38327
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 15:56:16.938405 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33569
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:56:17.262521 18818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:56:17.262789 18818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:56:17.262943 18818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:56:17.271733 18818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:56:17.271845 18818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:56:17.272033 18818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:56:17.282017 18818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33569
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:56:17.283336 18818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:56:17.284638 18818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:56:17.297887 18823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:17.297844 18824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:17.297912 18826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:56:17.299288 18818 server_base.cc:1034] running on GCE node
I20260929 15:56:17.299957 18818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:56:17.300907 18818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:56:17.302103 18818 hybrid_clock.cc:648] HybridClock initialized: now 1790697377302035 us; error 51 us; skew 500 ppm
I20260929 15:56:17.302380 18818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:56:17 dist-test-slave-r0ck krb5kdc[18578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:56:17.308930 18818 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:56:17.310482 18818 webserver.cc:460] Webserver started at http://127.0.97.2:46385/ using document root <none> and password file <none>
I20260929 15:56:17.311158 18818 fs_manager.cc:359] Metadata directory not provided
I20260929 15:56:17.311287 18818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:56:17.311638 18818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:56:17.314510 18818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-1/data/instance:
uuid: "52c12acd5d844a46bbec2fe81e134405"
format_stamp: "Formatted at 2026-09-29 15:56:17 on dist-test-slave-r0ck"
server_key: "f12b0ae3f1e6f9b433e10a8a4ab42d12"
server_key_iv: "d347ddb7c0198d0e65fad370470444ea"
server_key_version: "encryptionkey@0"
I20260929 15:56:17.315284 18818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "52c12acd5d844a46bbec2fe81e134405"
format_stamp: "Formatted at 2026-09-29 15:56:17 on dist-test-slave-r0ck"
server_key: "f12b0ae3f1e6f9b433e10a8a4ab42d12"
server_key_iv: "d347ddb7c0198d0e65fad370470444ea"
server_key_version: "encryptionkey@0"
I20260929 15:56:17.320138 18818 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 15:56:17.323539 18833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:17.324870 18818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 15:56:17.325063 18818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-1/wal
uuid: "52c12acd5d844a46bbec2fe81e134405"
format_stamp: "Formatted at 2026-09-29 15:56:17 on dist-test-slave-r0ck"
server_key: "f12b0ae3f1e6f9b433e10a8a4ab42d12"
server_key_iv: "d347ddb7c0198d0e65fad370470444ea"
server_key_version: "encryptionkey@0"
I20260929 15:56:17.325263 18818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:56:17.341305 18818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:56:17.345831 18818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:56:17.346122 18818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:56:17.347091 18818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:56:17.348536 18818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:56:17.348635 18818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:17.348790 18818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:56:17.348868 18818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:17.386358 18818 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38149
I20260929 15:56:17.386366 18945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38149 every 8 connection(s)
I20260929 15:56:17.387624 18818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:56:17.393432 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 18818
I20260929 15:56:17.393630 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:56:17.394536 388 external_mini_cluster.cc:1428] Setting key db0120c9dbccd39e19cb20a0609e0738
Sep 29 15:56:17 dist-test-slave-r0ck krb5kdc[18578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:56:17.411964 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:17.390281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34915 (local address 127.0.97.62:33569)
0929 15:56:17.390728 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:17.390735 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:17.391672 (+ 937us) server_negotiation.cc:407] Connection header received
0929 15:56:17.394254 (+ 2582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:17.394258 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:17.394330 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:17.394448 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:17.396239 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:17.397155 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:17.400204 (+ 3049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:17.400732 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:17.400839 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:17.405388 (+ 4549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:17.405396 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:17.405401 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:56:17.405441 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:17.408057 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:17.408823 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:17.408831 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:17.408838 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:17.409018 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:17.409620 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:17.409628 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:17.409635 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:17.410010 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:17.410180 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:17.411175 (+ 995us) server_negotiation.cc:299] Negotiation successful
0929 15:56:17.411433 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":316,"thread_start_us":210,"threads_started":1}
I20260929 15:56:17.412883 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:17.390822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33569 (local address 127.0.97.2:34915)
0929 15:56:17.391513 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:17.391556 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:17.393631 (+ 2075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:17.394721 (+ 1090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:17.394733 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:17.395388 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:56:17.396029 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:17.396054 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:17.397323 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:17.397328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:17.399996 (+ 2668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:17.400014 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:17.400969 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:17.400978 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:17.401436 (+ 458us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:17.402184 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:56:17.402219 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:56:17.405098 (+ 2879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:56:17.408214 (+ 3116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:56:17.408223 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:56:17.408237 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:56:17.408665 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:56:17.409159 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:56:17.409168 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:56:17.409174 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:56:17.409469 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:56:17.410210 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:56:17.410221 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:56:17.410676 (+ 455us) client_negotiation.cc:769] Sending connection context
0929 15:56:17.410993 (+ 317us) client_negotiation.cc:240] Negotiation successful
0929 15:56:17.411538 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":282,"threads_started":1}
I20260929 15:56:17.414680 18946 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33569
I20260929 15:56:17.415226 18946 heartbeater.cc:464] Registering TS with master...
I20260929 15:56:17.416375 18946 heartbeater.cc:511] Master 127.0.97.62:33569 requested a full tablet report, sending...
I20260929 15:56:17.418038 18623 ts_manager.cc:194] Registered new tserver with Master: 52c12acd5d844a46bbec2fe81e134405 (127.0.97.2:38149)
I20260929 15:56:17.418898 18623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34915
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 15:56:17.453084 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33569
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:56:17.779350 18952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:56:17.779656 18952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:56:17.779750 18952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:56:17.787091 18952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:56:17.787195 18952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:56:17.787359 18952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:56:17.799674 18952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33569
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:56:17.801072 18952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:56:17.803093 18952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:56:17.820067 18957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:17.820178 18958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:17.820426 18960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:56:17.821161 18952 server_base.cc:1034] running on GCE node
I20260929 15:56:17.821821 18952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:56:17.822887 18952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:56:17.824146 18952 hybrid_clock.cc:648] HybridClock initialized: now 1790697377824095 us; error 46 us; skew 500 ppm
I20260929 15:56:17.824494 18952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:56:17 dist-test-slave-r0ck krb5kdc[18578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:56:17.827636 18952 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:56:17.829102 18952 webserver.cc:460] Webserver started at http://127.0.97.3:43697/ using document root <none> and password file <none>
I20260929 15:56:17.829875 18952 fs_manager.cc:359] Metadata directory not provided
I20260929 15:56:17.830035 18952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:56:17.830374 18952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:56:17.833190 18952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-2/data/instance:
uuid: "0b61886d8169429ab1664dfbab936d61"
format_stamp: "Formatted at 2026-09-29 15:56:17 on dist-test-slave-r0ck"
server_key: "6cd0ddc711c7f58d764e9b33b34912e5"
server_key_iv: "5c60ce927edc8c52bd073470d5d00a6a"
server_key_version: "encryptionkey@0"
I20260929 15:56:17.834038 18952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "0b61886d8169429ab1664dfbab936d61"
format_stamp: "Formatted at 2026-09-29 15:56:17 on dist-test-slave-r0ck"
server_key: "6cd0ddc711c7f58d764e9b33b34912e5"
server_key_iv: "5c60ce927edc8c52bd073470d5d00a6a"
server_key_version: "encryptionkey@0"
I20260929 15:56:17.839316 18952 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260929 15:56:17.842957 18967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:17.844228 18952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 15:56:17.844398 18952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-2/wal
uuid: "0b61886d8169429ab1664dfbab936d61"
format_stamp: "Formatted at 2026-09-29 15:56:17 on dist-test-slave-r0ck"
server_key: "6cd0ddc711c7f58d764e9b33b34912e5"
server_key_iv: "5c60ce927edc8c52bd073470d5d00a6a"
server_key_version: "encryptionkey@0"
I20260929 15:56:17.844594 18952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:56:17.866747 18952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:56:17.871308 18952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:56:17.871614 18952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:56:17.872596 18952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:56:17.874019 18952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:56:17.874089 18952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:17.874261 18952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:56:17.874310 18952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:17.902137 18812 heartbeater.cc:503] Master 127.0.97.62:33569 was elected leader, sending a full tablet report...
I20260929 15:56:17.908526 18952 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43403
I20260929 15:56:17.908612 19079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43403 every 8 connection(s)
I20260929 15:56:17.909791 18952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:56:17.917644 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 18952
I20260929 15:56:17.917820 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:56:17.918322 388 external_mini_cluster.cc:1428] Setting key 46faf7ed3beddfa75c64b119996338cf
Sep 29 15:56:17 dist-test-slave-r0ck krb5kdc[18578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:56:17.929710 19082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:17.912369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49283 (local address 127.0.97.62:33569)
0929 15:56:17.913086 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:17.913093 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:17.913639 (+ 546us) server_negotiation.cc:407] Connection header received
0929 15:56:17.915452 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:17.915457 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:17.915550 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:17.915666 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:17.917107 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:17.918275 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:17.920077 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:17.920698 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:17.920825 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:17.924244 (+ 3419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:17.924249 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:17.924252 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:56:17.924280 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:17.926827 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:17.927497 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:17.927501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:17.927503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:17.927608 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:17.927984 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:17.927987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:17.927989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:17.928328 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:17.928450 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:17.929133 (+ 683us) server_negotiation.cc:299] Negotiation successful
0929 15:56:17.929429 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":377,"threads_started":1}
I20260929 15:56:17.930619 19083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:17.912578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33569 (local address 127.0.97.3:49283)
0929 15:56:17.913480 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:17.913522 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:17.915185 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:17.915882 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:17.915890 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:17.916377 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:56:17.916950 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:17.916968 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:17.918447 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:17.918453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:17.919937 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:17.919954 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:17.920923 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:17.920934 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:17.921192 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:17.921889 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:56:17.921911 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:56:17.924053 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:56:17.926970 (+ 2917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:56:17.926976 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:56:17.926987 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:56:17.927392 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:56:17.927699 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:56:17.927702 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:56:17.927705 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:56:17.927888 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:56:17.928450 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:56:17.928456 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:56:17.928759 (+ 303us) client_negotiation.cc:769] Sending connection context
0929 15:56:17.929007 (+ 248us) client_negotiation.cc:240] Negotiation successful
0929 15:56:17.929514 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":353,"threads_started":1}
I20260929 15:56:17.932134 19080 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33569
I20260929 15:56:17.932492 19080 heartbeater.cc:464] Registering TS with master...
I20260929 15:56:17.933238 19080 heartbeater.cc:511] Master 127.0.97.62:33569 requested a full tablet report, sending...
I20260929 15:56:17.934598 18623 ts_manager.cc:194] Registered new tserver with Master: 0b61886d8169429ab1664dfbab936d61 (127.0.97.3:43403)
I20260929 15:56:17.935401 18623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:49283
I20260929 15:56:17.945616 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:56:17.962774 19082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:17.949800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38884 (local address 127.0.97.62:33569)
0929 15:56:17.950023 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:17.950030 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:17.950292 (+ 262us) server_negotiation.cc:407] Connection header received
0929 15:56:17.950485 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:17.950490 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:17.950560 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:17.950661 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:17.952068 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:17.952794 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:17.954129 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:17.954552 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:17.954689 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:17.957326 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:17.957331 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:17.957334 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:56:17.957418 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:17.959863 (+ 2445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:17.960650 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:17.960656 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:17.960660 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:17.960780 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:17.961242 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:17.961247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:17.961250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:17.961632 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:17.961917 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:17.962284 (+ 367us) server_negotiation.cc:299] Negotiation successful
0929 15:56:17.962476 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 15:56:17.966678 18623 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38884:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:56:17.973804 18623 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:56:17.998858 19096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:17.986484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52144 (local address 127.0.97.2:38149)
0929 15:56:17.988766 (+ 2282us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:17.988779 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:17.990616 (+ 1837us) server_negotiation.cc:407] Connection header received
0929 15:56:17.990710 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:17.990717 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:17.990930 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:17.991124 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:56:17.991986 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:17.993463 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:17.995995 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:17.997327 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:17.997520 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:17.997718 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:17.998368 (+ 650us) server_negotiation.cc:299] Negotiation successful
0929 15:56:17.998536 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2058,"thread_start_us":1925,"threads_started":1}
I20260929 15:56:17.998878 19094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:17.985922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38149 (local address 127.0.0.1:52144)
0929 15:56:17.990193 (+ 4271us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:17.990321 (+ 128us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:17.990468 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:17.991369 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:17.991372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:17.991396 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:56:17.991785 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:17.991800 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:17.993597 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:17.993603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:17.995846 (+ 2243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:17.995858 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:17.997625 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:17.997635 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:17.997873 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:17.997941 (+ 68us) client_negotiation.cc:769] Sending connection context
0929 15:56:17.998204 (+ 263us) client_negotiation.cc:240] Negotiation successful
0929 15:56:17.998428 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4140,"mutex_wait_us":24,"thread_start_us":3881,"threads_started":1}
I20260929 15:56:17.999989 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:17.985673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35355 (local address 127.0.0.1:51392)
0929 15:56:17.987683 (+ 2010us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:17.987741 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:17.988009 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:17.990670 (+ 2661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:17.990677 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:17.990716 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:56:17.991217 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:17.991231 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:17.993610 (+ 2379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:17.993617 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:17.996255 (+ 2638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:17.996276 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:17.998120 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:17.998127 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:17.998362 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:17.998400 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 15:56:17.999458 (+ 1058us) client_negotiation.cc:240] Negotiation successful
0929 15:56:17.999598 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1685,"mutex_wait_us":1,"thread_start_us":1515,"threads_started":1}
I20260929 15:56:18.000397 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:17.985930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51392 (local address 127.0.97.1:35355)
0929 15:56:17.989901 (+ 3971us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:17.989910 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:17.989943 (+ 33us) server_negotiation.cc:407] Connection header received
0929 15:56:17.990047 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:17.990057 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:17.990434 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:17.991399 (+ 965us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:56:17.991656 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:17.993297 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:17.996462 (+ 3165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:17.997898 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:17.998045 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:17.998255 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:17.999782 (+ 1527us) server_negotiation.cc:299] Negotiation successful
0929 15:56:17.999964 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3818,"thread_start_us":3651,"threads_started":1}
I20260929 15:56:18.000718 19098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:17.988765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43403 (local address 127.0.0.1:50828)
0929 15:56:17.990825 (+ 2060us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:17.990847 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:17.990978 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:17.992339 (+ 1361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:17.992345 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:17.992379 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:56:17.992952 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:17.992967 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:17.995347 (+ 2380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:17.995353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:17.997474 (+ 2121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:17.997492 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:17.999872 (+ 2380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:17.999879 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:18.000092 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:18.000125 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 15:56:18.000221 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 15:56:18.000405 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1715,"thread_start_us":1362,"threads_started":1}
I20260929 15:56:18.001432 19097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:17.989516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50828 (local address 127.0.97.3:43403)
0929 15:56:17.991505 (+ 1989us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:17.991513 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:17.991543 (+ 30us) server_negotiation.cc:407] Connection header received
0929 15:56:17.991637 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:17.991643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:17.992052 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:17.992344 (+ 292us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:56:17.993179 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:17.995112 (+ 1933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:17.997707 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:17.999378 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:17.999525 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:17.999722 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:18.000885 (+ 1163us) server_negotiation.cc:299] Negotiation successful
0929 15:56:18.001059 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1821,"thread_start_us":2135,"threads_started":1}
I20260929 15:56:18.003610 18881 tablet_service.cc:1467] Processing CreateTablet for tablet b0524fb330bd4c24a350cfa5963ca87a (DEFAULT_TABLE table=test-table [id=001c8a3be34448f5ba15f9332f87b092]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:56:18.004348 18747 tablet_service.cc:1467] Processing CreateTablet for tablet b0524fb330bd4c24a350cfa5963ca87a (DEFAULT_TABLE table=test-table [id=001c8a3be34448f5ba15f9332f87b092]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:56:18.004611 19015 tablet_service.cc:1467] Processing CreateTablet for tablet b0524fb330bd4c24a350cfa5963ca87a (DEFAULT_TABLE table=test-table [id=001c8a3be34448f5ba15f9332f87b092]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:56:18.005165 18881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0524fb330bd4c24a350cfa5963ca87a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:56:18.005654 18747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0524fb330bd4c24a350cfa5963ca87a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:56:18.005911 19015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0524fb330bd4c24a350cfa5963ca87a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:56:18.020076 19099 tablet_bootstrap.cc:492] T b0524fb330bd4c24a350cfa5963ca87a P 0b61886d8169429ab1664dfbab936d61: Bootstrap starting.
I20260929 15:56:18.023339 19099 tablet_bootstrap.cc:654] T b0524fb330bd4c24a350cfa5963ca87a P 0b61886d8169429ab1664dfbab936d61: Neither blocks nor log segments found. Creating new log.
I20260929 15:56:18.024190 19101 tablet_bootstrap.cc:492] T b0524fb330bd4c24a350cfa5963ca87a P 12070c00f03f468b8ad0e99a6a36b073: Bootstrap starting.
I20260929 15:56:18.024668 19099 log.cc:824] T b0524fb330bd4c24a350cfa5963ca87a P 0b61886d8169429ab1664dfbab936d61: Log is configured to *not* fsync() on all Append() calls
I20260929 15:56:18.026042 19100 tablet_bootstrap.cc:492] T b0524fb330bd4c24a350cfa5963ca87a P 52c12acd5d844a46bbec2fe81e134405: Bootstrap starting.
I20260929 15:56:18.028779 19101 tablet_bootstrap.cc:654] T b0524fb330bd4c24a350cfa5963ca87a P 12070c00f03f468b8ad0e99a6a36b073: Neither blocks nor log segments found. Creating new log.
I20260929 15:56:18.029381 19100 tablet_bootstrap.cc:654] T b0524fb330bd4c24a350cfa5963ca87a P 52c12acd5d844a46bbec2fe81e134405: Neither blocks nor log segments found. Creating new log.
I20260929 15:56:18.030175 19101 log.cc:824] T b0524fb330bd4c24a350cfa5963ca87a P 12070c00f03f468b8ad0e99a6a36b073: Log is configured to *not* fsync() on all Append() calls
I20260929 15:56:18.030624 19100 log.cc:824] T b0524fb330bd4c24a350cfa5963ca87a P 52c12acd5d844a46bbec2fe81e134405: Log is configured to *not* fsync() on all Append() calls
I20260929 15:56:18.032748 19099 tablet_bootstrap.cc:492] T b0524fb330bd4c24a350cfa5963ca87a P 0b61886d8169429ab1664dfbab936d61: No bootstrap required, opened a new log
I20260929 15:56:18.033128 19099 ts_tablet_manager.cc:1397] T b0524fb330bd4c24a350cfa5963ca87a P 0b61886d8169429ab1664dfbab936d61: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20260929 15:56:18.034407 19100 tablet_bootstrap.cc:492] T b0524fb330bd4c24a350cfa5963ca87a P 52c12acd5d844a46bbec2fe81e134405: No bootstrap required, opened a new log
I20260929 15:56:18.034667 19100 ts_tablet_manager.cc:1397] T b0524fb330bd4c24a350cfa5963ca87a P 52c12acd5d844a46bbec2fe81e134405: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260929 15:56:18.035988 19101 tablet_bootstrap.cc:492] T b0524fb330bd4c24a350cfa5963ca87a P 12070c00f03f468b8ad0e99a6a36b073: No bootstrap required, opened a new log
I20260929 15:56:18.036238 19101 ts_tablet_manager.cc:1397] T b0524fb330bd4c24a350cfa5963ca87a P 12070c00f03f468b8ad0e99a6a36b073: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260929 15:56:18.042309 19100 raft_consensus.cc:348] T b0524fb330bd4c24a350cfa5963ca87a P 52c12acd5d844a46bbec2fe81e134405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12070c00f03f468b8ad0e99a6a36b073" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35355 } } peers { permanent_uuid: "52c12acd5d844a46bbec2fe81e134405" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38149 } } peers { permanent_uuid: "0b61886d8169429ab1664dfbab936d61" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43403 } }
I20260929 15:56:18.042697 19100 raft_consensus.cc:374] T b0524fb330bd4c24a350cfa5963ca87a P 52c12acd5d844a46bbec2fe81e134405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:56:18.042788 19100 raft_consensus.cc:729] T b0524fb330bd4c24a350cfa5963ca87a P 52c12acd5d844a46bbec2fe81e134405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c12acd5d844a46bbec2fe81e134405, State: Initialized, Role: FOLLOWER
I20260929 15:56:18.043403 19100 consensus_queue.cc:260] T b0524fb330bd4c24a350cfa5963ca87a P 52c12acd5d844a46bbec2fe81e134405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12070c00f03f468b8ad0e99a6a36b073" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35355 } } peers { permanent_uuid: "52c12acd5d844a46bbec2fe81e134405" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38149 } } peers { permanent_uuid: "0b61886d8169429ab1664dfbab936d61" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43403 } }
I20260929 15:56:18.044523 18946 heartbeater.cc:503] Master 127.0.97.62:33569 was elected leader, sending a full tablet report...
I20260929 15:56:18.044739 19099 raft_consensus.cc:348] T b0524fb330bd4c24a350cfa5963ca87a P 0b61886d8169429ab1664dfbab936d61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12070c00f03f468b8ad0e99a6a36b073" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35355 } } peers { permanent_uuid: "52c12acd5d844a46bbec2fe81e134405" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38149 } } peers { permanent_uuid: "0b61886d8169429ab1664dfbab936d61" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43403 } }
I20260929 15:56:18.045197 19100 ts_tablet_manager.cc:1428] T b0524fb330bd4c24a350cfa5963ca87a P 52c12acd5d844a46bbec2fe81e134405: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260929 15:56:18.045382 19099 raft_consensus.cc:374] T b0524fb330bd4c24a350cfa5963ca87a P 0b61886d8169429ab1664dfbab936d61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:56:18.045583 19099 raft_consensus.cc:729] T b0524fb330bd4c24a350cfa5963ca87a P 0b61886d8169429ab1664dfbab936d61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b61886d8169429ab1664dfbab936d61, State: Initialized, Role: FOLLOWER
I20260929 15:56:18.046725 19099 consensus_queue.cc:260] T b0524fb330bd4c24a350cfa5963ca87a P 0b61886d8169429ab1664dfbab936d61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12070c00f03f468b8ad0e99a6a36b073" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35355 } } peers { permanent_uuid: "52c12acd5d844a46bbec2fe81e134405" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38149 } } peers { permanent_uuid: "0b61886d8169429ab1664dfbab936d61" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43403 } }
I20260929 15:56:18.048290 19080 heartbeater.cc:503] Master 127.0.97.62:33569 was elected leader, sending a full tablet report...
I20260929 15:56:18.048091 19101 raft_consensus.cc:348] T b0524fb330bd4c24a350cfa5963ca87a P 12070c00f03f468b8ad0e99a6a36b073 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12070c00f03f468b8ad0e99a6a36b073" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35355 } } peers { permanent_uuid: "52c12acd5d844a46bbec2fe81e134405" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38149 } } peers { permanent_uuid: "0b61886d8169429ab1664dfbab936d61" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43403 } }
I20260929 15:56:18.048585 19101 raft_consensus.cc:374] T b0524fb330bd4c24a350cfa5963ca87a P 12070c00f03f468b8ad0e99a6a36b073 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:56:18.048692 19101 raft_consensus.cc:729] T b0524fb330bd4c24a350cfa5963ca87a P 12070c00f03f468b8ad0e99a6a36b073 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12070c00f03f468b8ad0e99a6a36b073, State: Initialized, Role: FOLLOWER
I20260929 15:56:18.049290 19099 ts_tablet_manager.cc:1428] T b0524fb330bd4c24a350cfa5963ca87a P 0b61886d8169429ab1664dfbab936d61: Time spent starting tablet: real 0.016s user 0.017s sys 0.001s
I20260929 15:56:18.049408 19101 consensus_queue.cc:260] T b0524fb330bd4c24a350cfa5963ca87a P 12070c00f03f468b8ad0e99a6a36b073 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12070c00f03f468b8ad0e99a6a36b073" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35355 } } peers { permanent_uuid: "52c12acd5d844a46bbec2fe81e134405" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38149 } } peers { permanent_uuid: "0b61886d8169429ab1664dfbab936d61" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43403 } }
I20260929 15:56:18.052495 19101 ts_tablet_manager.cc:1428] T b0524fb330bd4c24a350cfa5963ca87a P 12070c00f03f468b8ad0e99a6a36b073: Time spent starting tablet: real 0.016s user 0.016s sys 0.001s
W20260929 15:56:18.116266 18813 tablet.cc:2369] T b0524fb330bd4c24a350cfa5963ca87a P 12070c00f03f468b8ad0e99a6a36b073: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:56:18.139699 18947 tablet.cc:2369] T b0524fb330bd4c24a350cfa5963ca87a P 52c12acd5d844a46bbec2fe81e134405: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:56:18.140162 19106 raft_consensus.cc:482] T b0524fb330bd4c24a350cfa5963ca87a P 0b61886d8169429ab1664dfbab936d61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:56:18.140478 19106 raft_consensus.cc:504] T b0524fb330bd4c24a350cfa5963ca87a P 0b61886d8169429ab1664dfbab936d61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12070c00f03f468b8ad0e99a6a36b073" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35355 } } peers { permanent_uuid: "52c12acd5d844a46bbec2fe81e134405" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38149 } } peers { permanent_uuid: "0b61886d8169429ab1664dfbab936d61" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43403 } }
I20260929 15:56:18.142328 19106 leader_election.cc:288] T b0524fb330bd4c24a350cfa5963ca87a P 0b61886d8169429ab1664dfbab936d61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12070c00f03f468b8ad0e99a6a36b073 (127.0.97.1:35355), 52c12acd5d844a46bbec2fe81e134405 (127.0.97.2:38149)
I20260929 15:56:18.148578 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:18.142743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41939 (local address 127.0.97.1:35355)
0929 15:56:18.142963 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:18.142969 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:18.143268 (+ 299us) server_negotiation.cc:407] Connection header received
0929 15:56:18.143448 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:18.143455 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:18.143536 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:18.143624 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:56:18.144197 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:18.145111 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:18.146874 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:18.147713 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:18.147863 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:18.147949 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:18.148296 (+ 347us) server_negotiation.cc:299] Negotiation successful
0929 15:56:18.148391 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 15:56:18.148685 19083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:18.142754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35355 (local address 127.0.97.3:41939)
0929 15:56:18.143113 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:18.143132 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:18.143272 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:18.143738 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:18.143743 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:18.143775 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:56:18.144022 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:18.144032 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:18.145244 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:18.145252 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:18.146711 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:18.146725 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:18.147893 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:18.147900 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:18.148098 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:18.148134 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 15:56:18.148254 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 15:56:18.148396 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233}
I20260929 15:56:18.149677 19096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:18.142727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56871 (local address 127.0.97.2:38149)
0929 15:56:18.142954 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:18.142960 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:18.143745 (+ 785us) server_negotiation.cc:407] Connection header received
0929 15:56:18.144045 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:18.144050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:18.144121 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:18.144226 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:56:18.144765 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:18.145520 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:18.147490 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:18.148746 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:18.148881 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:18.148982 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:18.149273 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 15:56:18.149439 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 15:56:18.149783 18767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0524fb330bd4c24a350cfa5963ca87a" candidate_uuid: "0b61886d8169429ab1664dfbab936d61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12070c00f03f468b8ad0e99a6a36b073" is_pre_election: true
I20260929 15:56:18.150053 19108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:18.142832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38149 (local address 127.0.97.3:56871)
0929 15:56:18.143646 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:18.143666 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:18.143887 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:18.144240 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:18.144244 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:18.144273 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:56:18.144576 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:18.144587 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:18.145668 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:18.145673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:18.147338 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:18.147351 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:18.148960 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:18.148966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:18.149117 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:18.149139 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 15:56:18.149245 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 15:56:18.149383 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"mutex_wait_us":1,"thread_start_us":653,"threads_started":1}
I20260929 15:56:18.150444 18767 raft_consensus.cc:2454] T b0524fb330bd4c24a350cfa5963ca87a P 12070c00f03f468b8ad0e99a6a36b073 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b61886d8169429ab1664dfbab936d61 in term 0.
I20260929 15:56:18.151016 18901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0524fb330bd4c24a350cfa5963ca87a" candidate_uuid: "0b61886d8169429ab1664dfbab936d61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c12acd5d844a46bbec2fe81e134405" is_pre_election: true
I20260929 15:56:18.151218 18970 leader_election.cc:302] T b0524fb330bd4c24a350cfa5963ca87a P 0b61886d8169429ab1664dfbab936d61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b61886d8169429ab1664dfbab936d61, 12070c00f03f468b8ad0e99a6a36b073; no voters:
I20260929 15:56:18.151492 18901 raft_consensus.cc:2454] T b0524fb330bd4c24a350cfa5963ca87a P 52c12acd5d844a46bbec2fe81e134405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b61886d8169429ab1664dfbab936d61 in term 0.
I20260929 15:56:18.151672 19106 raft_consensus.cc:2781] T b0524fb330bd4c24a350cfa5963ca87a P 0b61886d8169429ab1664dfbab936d61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:56:18.151818 19106 raft_consensus.cc:482] T b0524fb330bd4c24a350cfa5963ca87a P 0b61886d8169429ab1664dfbab936d61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:56:18.151955 19106 raft_consensus.cc:3037] T b0524fb330bd4c24a350cfa5963ca87a P 0b61886d8169429ab1664dfbab936d61 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:56:18.154330 19106 raft_consensus.cc:504] T b0524fb330bd4c24a350cfa5963ca87a P 0b61886d8169429ab1664dfbab936d61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12070c00f03f468b8ad0e99a6a36b073" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35355 } } peers { permanent_uuid: "52c12acd5d844a46bbec2fe81e134405" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38149 } } peers { permanent_uuid: "0b61886d8169429ab1664dfbab936d61" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43403 } }
I20260929 15:56:18.154911 19106 leader_election.cc:288] T b0524fb330bd4c24a350cfa5963ca87a P 0b61886d8169429ab1664dfbab936d61 [CANDIDATE]: Term 1 election: Requested vote from peers 12070c00f03f468b8ad0e99a6a36b073 (127.0.97.1:35355), 52c12acd5d844a46bbec2fe81e134405 (127.0.97.2:38149)
I20260929 15:56:18.155387 18901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0524fb330bd4c24a350cfa5963ca87a" candidate_uuid: "0b61886d8169429ab1664dfbab936d61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c12acd5d844a46bbec2fe81e134405"
I20260929 15:56:18.155385 18767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0524fb330bd4c24a350cfa5963ca87a" candidate_uuid: "0b61886d8169429ab1664dfbab936d61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12070c00f03f468b8ad0e99a6a36b073"
I20260929 15:56:18.155599 18901 raft_consensus.cc:3037] T b0524fb330bd4c24a350cfa5963ca87a P 52c12acd5d844a46bbec2fe81e134405 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:56:18.155599 18767 raft_consensus.cc:3037] T b0524fb330bd4c24a350cfa5963ca87a P 12070c00f03f468b8ad0e99a6a36b073 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:56:18.159005 18901 raft_consensus.cc:2454] T b0524fb330bd4c24a350cfa5963ca87a P 52c12acd5d844a46bbec2fe81e134405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b61886d8169429ab1664dfbab936d61 in term 1.
I20260929 15:56:18.159005 18767 raft_consensus.cc:2454] T b0524fb330bd4c24a350cfa5963ca87a P 12070c00f03f468b8ad0e99a6a36b073 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b61886d8169429ab1664dfbab936d61 in term 1.
I20260929 15:56:18.159454 18970 leader_election.cc:302] T b0524fb330bd4c24a350cfa5963ca87a P 0b61886d8169429ab1664dfbab936d61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b61886d8169429ab1664dfbab936d61, 12070c00f03f468b8ad0e99a6a36b073; no voters:
I20260929 15:56:18.159772 19106 raft_consensus.cc:2781] T b0524fb330bd4c24a350cfa5963ca87a P 0b61886d8169429ab1664dfbab936d61 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:56:18.160274 19106 raft_consensus.cc:686] T b0524fb330bd4c24a350cfa5963ca87a P 0b61886d8169429ab1664dfbab936d61 [term 1 LEADER]: Becoming Leader. State: Replica: 0b61886d8169429ab1664dfbab936d61, State: Running, Role: LEADER
I20260929 15:56:18.160739 19106 consensus_queue.cc:237] T b0524fb330bd4c24a350cfa5963ca87a P 0b61886d8169429ab1664dfbab936d61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12070c00f03f468b8ad0e99a6a36b073" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35355 } } peers { permanent_uuid: "52c12acd5d844a46bbec2fe81e134405" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38149 } } peers { permanent_uuid: "0b61886d8169429ab1664dfbab936d61" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43403 } }
W20260929 15:56:18.161406 19081 tablet.cc:2369] T b0524fb330bd4c24a350cfa5963ca87a P 0b61886d8169429ab1664dfbab936d61: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:56:18.165206 18622 catalog_manager.cc:5495] T b0524fb330bd4c24a350cfa5963ca87a P 0b61886d8169429ab1664dfbab936d61 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b61886d8169429ab1664dfbab936d61 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "0b61886d8169429ab1664dfbab936d61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12070c00f03f468b8ad0e99a6a36b073" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52c12acd5d844a46bbec2fe81e134405" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b61886d8169429ab1664dfbab936d61" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43403 } health_report { overall_health: HEALTHY } } }
I20260929 15:56:18.193200 19097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:18.187775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50842 (local address 127.0.97.3:43403)
0929 15:56:18.187980 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:18.187987 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:18.188065 (+ 78us) server_negotiation.cc:407] Connection header received
0929 15:56:18.188328 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:18.188332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:18.188408 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:18.188560 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:56:18.188956 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:18.189694 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:18.191060 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:18.191451 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:18.191586 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:18.192148 (+ 562us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:56:18.192612 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:56:18.192807 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:18.192922 (+ 115us) server_negotiation.cc:299] Negotiation successful
0929 15:56:18.192966 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 15:56:18.200794 18901 raft_consensus.cc:1261] T b0524fb330bd4c24a350cfa5963ca87a P 52c12acd5d844a46bbec2fe81e134405 [term 1 FOLLOWER]: Refusing update from remote peer 0b61886d8169429ab1664dfbab936d61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:56:18.200862 18767 raft_consensus.cc:1261] T b0524fb330bd4c24a350cfa5963ca87a P 12070c00f03f468b8ad0e99a6a36b073 [term 1 FOLLOWER]: Refusing update from remote peer 0b61886d8169429ab1664dfbab936d61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:56:18.201763 19109 consensus_queue.cc:1035] T b0524fb330bd4c24a350cfa5963ca87a P 0b61886d8169429ab1664dfbab936d61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12070c00f03f468b8ad0e99a6a36b073" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:56:18.202071 19106 consensus_queue.cc:1035] T b0524fb330bd4c24a350cfa5963ca87a P 0b61886d8169429ab1664dfbab936d61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c12acd5d844a46bbec2fe81e134405" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:56:18.234277 18622 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:56:18.234565 18622 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:56:18.238862 18622 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0657ba45a321475bb0006b040ac5fc04: Sending DeleteTablet for 3 replicas of tablet b0524fb330bd4c24a350cfa5963ca87a
I20260929 15:56:18.240065 18747 tablet_service.cc:1514] Processing DeleteTablet for tablet b0524fb330bd4c24a350cfa5963ca87a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:56:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51392
I20260929 15:56:18.240413 19015 tablet_service.cc:1514] Processing DeleteTablet for tablet b0524fb330bd4c24a350cfa5963ca87a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:56:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50828
I20260929 15:56:18.240767 18881 tablet_service.cc:1514] Processing DeleteTablet for tablet b0524fb330bd4c24a350cfa5963ca87a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:56:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52144
I20260929 15:56:18.243438 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 18684
I20260929 15:56:18.249099 19123 tablet_replica.cc:330] T b0524fb330bd4c24a350cfa5963ca87a P 12070c00f03f468b8ad0e99a6a36b073: stopping tablet replica
I20260929 15:56:18.249735 19123 raft_consensus.cc:2229] T b0524fb330bd4c24a350cfa5963ca87a P 12070c00f03f468b8ad0e99a6a36b073 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:56:18.250386 19123 raft_consensus.cc:2258] T b0524fb330bd4c24a350cfa5963ca87a P 12070c00f03f468b8ad0e99a6a36b073 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:56:18.251291 19124 tablet_replica.cc:330] T b0524fb330bd4c24a350cfa5963ca87a P 0b61886d8169429ab1664dfbab936d61: stopping tablet replica
I20260929 15:56:18.251784 19124 raft_consensus.cc:2229] T b0524fb330bd4c24a350cfa5963ca87a P 0b61886d8169429ab1664dfbab936d61 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:56:18.252521 19124 raft_consensus.cc:2258] T b0524fb330bd4c24a350cfa5963ca87a P 0b61886d8169429ab1664dfbab936d61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:56:18.255645 19125 tablet_replica.cc:330] T b0524fb330bd4c24a350cfa5963ca87a P 52c12acd5d844a46bbec2fe81e134405: stopping tablet replica
I20260929 15:56:18.256336 19125 raft_consensus.cc:2229] T b0524fb330bd4c24a350cfa5963ca87a P 52c12acd5d844a46bbec2fe81e134405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:56:18.256858 19125 raft_consensus.cc:2258] T b0524fb330bd4c24a350cfa5963ca87a P 52c12acd5d844a46bbec2fe81e134405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:56:18.258917 19124 ts_tablet_manager.cc:1905] T b0524fb330bd4c24a350cfa5963ca87a P 0b61886d8169429ab1664dfbab936d61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:56:18.259341 19123 ts_tablet_manager.cc:1905] T b0524fb330bd4c24a350cfa5963ca87a P 12070c00f03f468b8ad0e99a6a36b073: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:56:18.266320 19125 ts_tablet_manager.cc:1905] T b0524fb330bd4c24a350cfa5963ca87a P 52c12acd5d844a46bbec2fe81e134405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:56:18.267061 19124 ts_tablet_manager.cc:1918] T b0524fb330bd4c24a350cfa5963ca87a P 0b61886d8169429ab1664dfbab936d61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:56:18.267279 19124 log.cc:1196] T b0524fb330bd4c24a350cfa5963ca87a P 0b61886d8169429ab1664dfbab936d61: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-2/wal/wals/b0524fb330bd4c24a350cfa5963ca87a
Sep 29 15:56:18 dist-test-slave-r0ck krb5kdc[18578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 15:56:18.267973 19124 ts_tablet_manager.cc:1939] T b0524fb330bd4c24a350cfa5963ca87a P 0b61886d8169429ab1664dfbab936d61: Deleting consensus metadata
I20260929 15:56:18.270035 18607 catalog_manager.cc:4841] TS 0b61886d8169429ab1664dfbab936d61 (127.0.97.3:43403): tablet b0524fb330bd4c24a350cfa5963ca87a (table test-table [id=001c8a3be34448f5ba15f9332f87b092]) successfully deleted
I20260929 15:56:18.272126 19123 ts_tablet_manager.cc:1918] T b0524fb330bd4c24a350cfa5963ca87a P 12070c00f03f468b8ad0e99a6a36b073: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:56:18.272368 19123 log.cc:1196] T b0524fb330bd4c24a350cfa5963ca87a P 12070c00f03f468b8ad0e99a6a36b073: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-0/wal/wals/b0524fb330bd4c24a350cfa5963ca87a
I20260929 15:56:18.272998 19123 ts_tablet_manager.cc:1939] T b0524fb330bd4c24a350cfa5963ca87a P 12070c00f03f468b8ad0e99a6a36b073: Deleting consensus metadata
I20260929 15:56:18.273940 19125 ts_tablet_manager.cc:1918] T b0524fb330bd4c24a350cfa5963ca87a P 52c12acd5d844a46bbec2fe81e134405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:56:18.274113 19125 log.cc:1196] T b0524fb330bd4c24a350cfa5963ca87a P 52c12acd5d844a46bbec2fe81e134405: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697075657932-388-0/minicluster-data/ts-1/wal/wals/b0524fb330bd4c24a350cfa5963ca87a
I20260929 15:56:18.274700 19125 ts_tablet_manager.cc:1939] T b0524fb330bd4c24a350cfa5963ca87a P 52c12acd5d844a46bbec2fe81e134405: Deleting consensus metadata
I20260929 15:56:18.275112 18609 catalog_manager.cc:4841] TS 12070c00f03f468b8ad0e99a6a36b073 (127.0.97.1:35355): tablet b0524fb330bd4c24a350cfa5963ca87a (table test-table [id=001c8a3be34448f5ba15f9332f87b092]) successfully deleted
I20260929 15:56:18.276239 18610 catalog_manager.cc:4841] TS 52c12acd5d844a46bbec2fe81e134405 (127.0.97.2:38149): tablet b0524fb330bd4c24a350cfa5963ca87a (table test-table [id=001c8a3be34448f5ba15f9332f87b092]) successfully deleted
I20260929 15:56:18.277117 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:18.244834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51396 (local address 127.0.97.1:35355)
0929 15:56:18.245187 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:18.245193 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:18.256672 (+ 11479us) server_negotiation.cc:407] Connection header received
0929 15:56:18.257130 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:18.257136 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:18.257264 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:18.257377 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:18.259270 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:18.260475 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:18.262473 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:18.263135 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:18.263252 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:18.268826 (+ 5574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:18.268841 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:18.268860 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:56:18.268905 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:18.273651 (+ 4746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:18.274617 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:18.274627 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:18.274630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:18.274795 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:18.275289 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:18.275295 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:18.275298 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:18.275751 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:18.276034 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:18.276611 (+ 577us) server_negotiation.cc:299] Negotiation successful
0929 15:56:18.276786 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260929 15:56:18.277851 18807 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:56:18.369307 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 18684
I20260929 15:56:18.392665 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 18818
Sep 29 15:56:18 dist-test-slave-r0ck krb5kdc[18578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 15:56:18.411654 19096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:18.393470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52154 (local address 127.0.97.2:38149)
0929 15:56:18.393662 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:18.393669 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:18.393766 (+ 97us) server_negotiation.cc:407] Connection header received
0929 15:56:18.393936 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:18.393942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:18.394030 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:18.394144 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:18.395758 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:18.396873 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:18.398711 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:18.399359 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:18.399477 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:18.404795 (+ 5318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:18.404809 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:18.404827 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:56:18.404867 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:18.408765 (+ 3898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:18.409614 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:18.409623 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:18.409627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:18.409772 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:18.410121 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:18.410127 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:18.410130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:18.410594 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:18.410884 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:18.411072 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 15:56:18.411299 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 15:56:18.412637 18941 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:56:18.531602 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 18818
I20260929 15:56:18.549940 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 18952
Sep 29 15:56:18 dist-test-slave-r0ck krb5kdc[18578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 15:56:18.568406 19097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:18.550669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50850 (local address 127.0.97.3:43403)
0929 15:56:18.550881 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:18.550887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:18.551002 (+ 115us) server_negotiation.cc:407] Connection header received
0929 15:56:18.551168 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:18.551174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:18.551259 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:18.551397 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:18.552582 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:18.553786 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:18.555037 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:18.555724 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:18.555853 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:18.560874 (+ 5021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:18.560888 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:18.560907 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:56:18.560957 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:18.564938 (+ 3981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:18.565781 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:18.565791 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:18.565794 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:18.565970 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:18.566334 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:18.566339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:18.566343 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:18.566829 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:18.567170 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:18.567747 (+ 577us) server_negotiation.cc:299] Negotiation successful
0929 15:56:18.567969 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 15:56:18.569206 19075 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:56:18.691658 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 18952
I20260929 15:56:18.711194 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 18591
I20260929 15:56:18.712205 18653 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:56:18.822577 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 18591
2026-09-29T15:56:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4427 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T15:56:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:56:18Z Disabled control of system clock
I20260929 15:56:18.899873 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46357
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46357
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 15:56:19.225742 19134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:56:19.226006 19134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:56:19.226091 19134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:56:19.233223 19134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:56:19.233333 19134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:56:19.233441 19134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:56:19.233505 19134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:56:19.233542 19134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:56:19.242573 19134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46357
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:56:19.243731 19134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:56:19.245401 19134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:56:19.256821 19139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:19.256783 19140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:19.257414 19142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:56:19.258311 19134 server_base.cc:1034] running on GCE node
I20260929 15:56:19.259035 19134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:56:19.260403 19134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:56:19.261648 19134 hybrid_clock.cc:648] HybridClock initialized: now 1790697379261587 us; error 44 us; skew 500 ppm
I20260929 15:56:19.261998 19134 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:56:19.264130 19134 webserver.cc:460] Webserver started at http://127.0.97.62:38837/ using document root <none> and password file <none>
I20260929 15:56:19.264797 19134 fs_manager.cc:359] Metadata directory not provided
I20260929 15:56:19.264948 19134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:56:19.265275 19134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:56:19.268041 19134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "07d068e6272248a38d07bdf311b1bf1e"
format_stamp: "Formatted at 2026-09-29 15:56:19 on dist-test-slave-r0ck"
server_key: "28526a3aa2e78b5b1e28e301a2ee59d1"
server_key_iv: "6aa32d136dcd6098a08dc585d01a314b"
server_key_version: "encryptionkey@0"
I20260929 15:56:19.268842 19134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "07d068e6272248a38d07bdf311b1bf1e"
format_stamp: "Formatted at 2026-09-29 15:56:19 on dist-test-slave-r0ck"
server_key: "28526a3aa2e78b5b1e28e301a2ee59d1"
server_key_iv: "6aa32d136dcd6098a08dc585d01a314b"
server_key_version: "encryptionkey@0"
I20260929 15:56:19.273813 19134 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 15:56:19.277537 19148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:19.278855 19134 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:56:19.279043 19134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "07d068e6272248a38d07bdf311b1bf1e"
format_stamp: "Formatted at 2026-09-29 15:56:19 on dist-test-slave-r0ck"
server_key: "28526a3aa2e78b5b1e28e301a2ee59d1"
server_key_iv: "6aa32d136dcd6098a08dc585d01a314b"
server_key_version: "encryptionkey@0"
I20260929 15:56:19.279289 19134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:56:19.306192 19134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:56:19.306999 19134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:56:19.307284 19134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:56:19.325649 19134 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46357
I20260929 15:56:19.325685 19199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46357 every 8 connection(s)
I20260929 15:56:19.326973 19134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:56:19.330552 19200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:56:19.331779 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 19134
I20260929 15:56:19.331964 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:56:19.332402 388 external_mini_cluster.cc:1428] Setting key 0278401088cda1713402c92b88c473fb
I20260929 15:56:19.342971 19200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07d068e6272248a38d07bdf311b1bf1e: Bootstrap starting.
I20260929 15:56:19.345491 19203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:19.334131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48458 (local address 127.0.97.62:46357)
0929 15:56:19.335237 (+ 1106us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:19.335255 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:19.335308 (+ 53us) server_negotiation.cc:407] Connection header received
0929 15:56:19.337006 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:19.337024 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:19.337518 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:19.337913 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:19.338441 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:19.339641 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:19.341251 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:19.341933 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:19.342124 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:19.343313 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:19.343320 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:19.343332 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:19.343848 (+ 516us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:19.343927 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:19.343946 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:19.344074 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:19.344303 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 15:56:19.344574 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":364,"threads_started":1}
I20260929 15:56:19.346237 19200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07d068e6272248a38d07bdf311b1bf1e: Neither blocks nor log segments found. Creating new log.
I20260929 15:56:19.347496 19200 log.cc:824] T 00000000000000000000000000000000 P 07d068e6272248a38d07bdf311b1bf1e: Log is configured to *not* fsync() on all Append() calls
I20260929 15:56:19.350267 19200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07d068e6272248a38d07bdf311b1bf1e: No bootstrap required, opened a new log
I20260929 15:56:19.357502 19200 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07d068e6272248a38d07bdf311b1bf1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07d068e6272248a38d07bdf311b1bf1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46357 } }
I20260929 15:56:19.357806 19200 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07d068e6272248a38d07bdf311b1bf1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:56:19.357879 19200 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07d068e6272248a38d07bdf311b1bf1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07d068e6272248a38d07bdf311b1bf1e, State: Initialized, Role: FOLLOWER
I20260929 15:56:19.358476 19200 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07d068e6272248a38d07bdf311b1bf1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07d068e6272248a38d07bdf311b1bf1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46357 } }
I20260929 15:56:19.358757 19200 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07d068e6272248a38d07bdf311b1bf1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:56:19.358901 19200 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07d068e6272248a38d07bdf311b1bf1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:56:19.359061 19200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07d068e6272248a38d07bdf311b1bf1e [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:56:19.361001 19200 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07d068e6272248a38d07bdf311b1bf1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07d068e6272248a38d07bdf311b1bf1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46357 } }
I20260929 15:56:19.361503 19200 leader_election.cc:302] T 00000000000000000000000000000000 P 07d068e6272248a38d07bdf311b1bf1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07d068e6272248a38d07bdf311b1bf1e; no voters:
I20260929 15:56:19.361989 19200 leader_election.cc:288] T 00000000000000000000000000000000 P 07d068e6272248a38d07bdf311b1bf1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:56:19.362134 19205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07d068e6272248a38d07bdf311b1bf1e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:56:19.362869 19205 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07d068e6272248a38d07bdf311b1bf1e [term 1 LEADER]: Becoming Leader. State: Replica: 07d068e6272248a38d07bdf311b1bf1e, State: Running, Role: LEADER
I20260929 15:56:19.363407 19205 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07d068e6272248a38d07bdf311b1bf1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07d068e6272248a38d07bdf311b1bf1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46357 } }
I20260929 15:56:19.363965 19200 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07d068e6272248a38d07bdf311b1bf1e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:56:19.366045 19206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07d068e6272248a38d07bdf311b1bf1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07d068e6272248a38d07bdf311b1bf1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07d068e6272248a38d07bdf311b1bf1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46357 } } }
I20260929 15:56:19.366067 19207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07d068e6272248a38d07bdf311b1bf1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07d068e6272248a38d07bdf311b1bf1e. Latest consensus state: current_term: 1 leader_uuid: "07d068e6272248a38d07bdf311b1bf1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07d068e6272248a38d07bdf311b1bf1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46357 } } }
I20260929 15:56:19.366490 19206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07d068e6272248a38d07bdf311b1bf1e [sys.catalog]: This master's current role is: LEADER
I20260929 15:56:19.366487 19207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07d068e6272248a38d07bdf311b1bf1e [sys.catalog]: This master's current role is: LEADER
I20260929 15:56:19.369477 19211 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:56:19.374604 19211 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:56:19.383462 19211 catalog_manager.cc:1347] Generated new cluster ID: 46de46154e3d47e79b0b02e85eca5b22
I20260929 15:56:19.383574 19211 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:56:19.425966 19211 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:56:19.427276 19211 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:56:19.435710 19211 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07d068e6272248a38d07bdf311b1bf1e: Generated new TSK 0
I20260929 15:56:19.436501 19211 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:56:19.447420 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46357
--builtin_ntp_servers=127.0.97.20:45455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 15:56:19.786399 19224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:56:19.786650 19224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:56:19.786736 19224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:56:19.793787 19224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:56:19.793880 19224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:56:19.793984 19224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:56:19.804591 19224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:56:19.806241 19224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:56:19.808596 19224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:56:19.821372 19230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:19.821413 19232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:19.821447 19229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:56:19.822563 19224 server_base.cc:1034] running on GCE node
I20260929 15:56:19.823135 19224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:56:19.824028 19224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:56:19.825289 19224 hybrid_clock.cc:648] HybridClock initialized: now 1790697379825242 us; error 47 us; skew 500 ppm
I20260929 15:56:19.825667 19224 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:56:19.827957 19224 webserver.cc:460] Webserver started at http://127.0.97.1:36079/ using document root <none> and password file <none>
I20260929 15:56:19.828639 19224 fs_manager.cc:359] Metadata directory not provided
I20260929 15:56:19.828770 19224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:56:19.829089 19224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:56:19.831830 19224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-0/data/instance:
uuid: "bac42ab8c24d46bab5406b255cafee3d"
format_stamp: "Formatted at 2026-09-29 15:56:19 on dist-test-slave-r0ck"
server_key: "bde715b7bf5b1d2b98635b3abafc6951"
server_key_iv: "04bc77a98a1bf062335f054a3afa15db"
server_key_version: "encryptionkey@0"
I20260929 15:56:19.832587 19224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "bac42ab8c24d46bab5406b255cafee3d"
format_stamp: "Formatted at 2026-09-29 15:56:19 on dist-test-slave-r0ck"
server_key: "bde715b7bf5b1d2b98635b3abafc6951"
server_key_iv: "04bc77a98a1bf062335f054a3afa15db"
server_key_version: "encryptionkey@0"
I20260929 15:56:19.837432 19224 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260929 15:56:19.840775 19238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:19.842106 19224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:56:19.842296 19224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-0/wal
uuid: "bac42ab8c24d46bab5406b255cafee3d"
format_stamp: "Formatted at 2026-09-29 15:56:19 on dist-test-slave-r0ck"
server_key: "bde715b7bf5b1d2b98635b3abafc6951"
server_key_iv: "04bc77a98a1bf062335f054a3afa15db"
server_key_version: "encryptionkey@0"
I20260929 15:56:19.842516 19224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:56:19.855880 19224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:56:19.856654 19224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:56:19.856992 19224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:56:19.857997 19224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:56:19.859437 19224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:56:19.859560 19224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:19.859701 19224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:56:19.859790 19224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:19.892692 19224 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40113
I20260929 15:56:19.892735 19350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40113 every 8 connection(s)
I20260929 15:56:19.894065 19224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:56:19.897867 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 19224
I20260929 15:56:19.898039 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:56:19.898638 388 external_mini_cluster.cc:1428] Setting key 97cd3f9d95713701b249711090d6437b
I20260929 15:56:19.901659 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46357
--builtin_ntp_servers=127.0.97.20:45455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 15:56:19.908174 19353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:19.896524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59545 (local address 127.0.97.62:46357)
0929 15:56:19.896991 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:19.896996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:19.898329 (+ 1333us) server_negotiation.cc:407] Connection header received
0929 15:56:19.900524 (+ 2195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:19.900529 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:19.900610 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:19.900713 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:19.902113 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:19.902813 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:19.905186 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:19.905654 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:19.905754 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:19.907427 (+ 1673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:19.907432 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:19.907435 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:19.907455 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:19.907493 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:19.907505 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:19.907577 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:19.907836 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 15:56:19.907911 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":241,"threads_started":1}
I20260929 15:56:19.909585 19354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:19.897079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46357 (local address 127.0.97.1:59545)
0929 15:56:19.898160 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:19.898215 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:19.900219 (+ 2004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:19.900987 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:19.900999 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:19.901104 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:56:19.901946 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:19.901973 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:19.902966 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:19.902975 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:19.905025 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:19.905042 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:19.905874 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:19.905883 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:19.906223 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:19.906991 (+ 768us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:56:19.906997 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:56:19.907050 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:56:19.907149 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:56:19.907607 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:56:19.907621 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:56:19.907632 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 15:56:19.907819 (+ 187us) client_negotiation.cc:240] Negotiation successful
0929 15:56:19.908234 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":347,"threads_started":1}
I20260929 15:56:19.911096 19351 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46357
I20260929 15:56:19.911420 19351 heartbeater.cc:464] Registering TS with master...
I20260929 15:56:19.912253 19351 heartbeater.cc:511] Master 127.0.97.62:46357 requested a full tablet report, sending...
I20260929 15:56:19.914414 19165 ts_manager.cc:194] Registered new tserver with Master: bac42ab8c24d46bab5406b255cafee3d (127.0.97.1:40113)
I20260929 15:56:19.915891 19165 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:59545
W20260929 15:56:20.229727 19355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:56:20.229990 19355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:56:20.230134 19355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:56:20.237555 19355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:56:20.237658 19355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:56:20.237846 19355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:56:20.247105 19355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:56:20.248404 19355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:56:20.250140 19355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:56:20.262915 19360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:20.262965 19361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:20.263237 19363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:56:20.264273 19355 server_base.cc:1034] running on GCE node
I20260929 15:56:20.264967 19355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:56:20.266022 19355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:56:20.267225 19355 hybrid_clock.cc:648] HybridClock initialized: now 1790697380267178 us; error 46 us; skew 500 ppm
I20260929 15:56:20.267555 19355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:56:20.269853 19355 webserver.cc:460] Webserver started at http://127.0.97.2:34777/ using document root <none> and password file <none>
I20260929 15:56:20.270525 19355 fs_manager.cc:359] Metadata directory not provided
I20260929 15:56:20.270650 19355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:56:20.270965 19355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:56:20.273746 19355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-1/data/instance:
uuid: "b91972eceda742b5a205e113ff6f32c4"
format_stamp: "Formatted at 2026-09-29 15:56:20 on dist-test-slave-r0ck"
server_key: "aa012d23ed4283df5a3fecee12d0b381"
server_key_iv: "9d4b00e92f2c16a3f18b8c3d1b787d9c"
server_key_version: "encryptionkey@0"
I20260929 15:56:20.274513 19355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b91972eceda742b5a205e113ff6f32c4"
format_stamp: "Formatted at 2026-09-29 15:56:20 on dist-test-slave-r0ck"
server_key: "aa012d23ed4283df5a3fecee12d0b381"
server_key_iv: "9d4b00e92f2c16a3f18b8c3d1b787d9c"
server_key_version: "encryptionkey@0"
I20260929 15:56:20.279903 19355 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 15:56:20.283692 19369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:20.285187 19355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 15:56:20.285406 19355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-1/wal
uuid: "b91972eceda742b5a205e113ff6f32c4"
format_stamp: "Formatted at 2026-09-29 15:56:20 on dist-test-slave-r0ck"
server_key: "aa012d23ed4283df5a3fecee12d0b381"
server_key_iv: "9d4b00e92f2c16a3f18b8c3d1b787d9c"
server_key_version: "encryptionkey@0"
I20260929 15:56:20.285648 19355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:56:20.300374 19355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:56:20.301553 19355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:56:20.301889 19355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:56:20.303121 19355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:56:20.305060 19355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:56:20.305202 19355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:20.305445 19355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:56:20.305548 19355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260929 15:56:20.344244 19355 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43635
I20260929 15:56:20.344271 19481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43635 every 8 connection(s)
I20260929 15:56:20.345600 19355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:56:20.349735 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 19355
I20260929 15:56:20.349902 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:56:20.350476 388 external_mini_cluster.cc:1428] Setting key 802b0709c768a9f57015c6c438fa99ab
I20260929 15:56:20.353585 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46357
--builtin_ntp_servers=127.0.97.20:45455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 15:56:20.359333 19353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:20.348307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53461 (local address 127.0.97.62:46357)
0929 15:56:20.348527 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:20.348533 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:20.350210 (+ 1677us) server_negotiation.cc:407] Connection header received
0929 15:56:20.352433 (+ 2223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:20.352439 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:20.352524 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:20.352639 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:20.353888 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:20.354582 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:20.356442 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:20.356839 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:20.356970 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:20.358500 (+ 1530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:20.358505 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:20.358508 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:20.358526 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:20.358565 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:20.358579 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:20.358684 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:20.358924 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 15:56:20.359024 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 15:56:20.360332 19484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:20.348887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46357 (local address 127.0.97.2:53461)
0929 15:56:20.350033 (+ 1146us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:20.350087 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:20.352139 (+ 2052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:20.352922 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:20.352930 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:20.353016 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:56:20.353622 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:20.353668 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:20.354743 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:20.354749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:20.356274 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:20.356290 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:20.357071 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:20.357081 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:20.357294 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:20.358143 (+ 849us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:56:20.358146 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:56:20.358185 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:56:20.358281 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:56:20.358682 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:56:20.358694 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:56:20.358704 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 15:56:20.358888 (+ 184us) client_negotiation.cc:240] Negotiation successful
0929 15:56:20.359283 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":328,"threads_started":1}
I20260929 15:56:20.361747 19482 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46357
I20260929 15:56:20.362061 19482 heartbeater.cc:464] Registering TS with master...
I20260929 15:56:20.362928 19482 heartbeater.cc:511] Master 127.0.97.62:46357 requested a full tablet report, sending...
I20260929 15:56:20.364480 19165 ts_manager.cc:194] Registered new tserver with Master: b91972eceda742b5a205e113ff6f32c4 (127.0.97.2:43635)
I20260929 15:56:20.365742 19165 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:53461
W20260929 15:56:20.706475 19485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:56:20.706740 19485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:56:20.706830 19485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:56:20.714069 19485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:56:20.714164 19485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:56:20.714268 19485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:56:20.723526 19485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:56:20.724751 19485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:56:20.726449 19485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:56:20.739348 19490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:20.739414 19493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:20.739367 19491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:56:20.740155 19485 server_base.cc:1034] running on GCE node
I20260929 15:56:20.740720 19485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:56:20.741696 19485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:56:20.742947 19485 hybrid_clock.cc:648] HybridClock initialized: now 1790697380742914 us; error 60 us; skew 500 ppm
I20260929 15:56:20.743264 19485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:56:20.745584 19485 webserver.cc:460] Webserver started at http://127.0.97.3:40489/ using document root <none> and password file <none>
I20260929 15:56:20.746253 19485 fs_manager.cc:359] Metadata directory not provided
I20260929 15:56:20.746383 19485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:56:20.746732 19485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:56:20.749547 19485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-2/data/instance:
uuid: "0cc103ecc96942db9f931e26677a35d4"
format_stamp: "Formatted at 2026-09-29 15:56:20 on dist-test-slave-r0ck"
server_key: "a329acf225ff29ccae763c66d479ce95"
server_key_iv: "32466b9106649e41fdb29c0f7bc1d327"
server_key_version: "encryptionkey@0"
I20260929 15:56:20.750336 19485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "0cc103ecc96942db9f931e26677a35d4"
format_stamp: "Formatted at 2026-09-29 15:56:20 on dist-test-slave-r0ck"
server_key: "a329acf225ff29ccae763c66d479ce95"
server_key_iv: "32466b9106649e41fdb29c0f7bc1d327"
server_key_version: "encryptionkey@0"
I20260929 15:56:20.755411 19485 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 15:56:20.758829 19499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:20.760172 19485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 15:56:20.760360 19485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-2/wal
uuid: "0cc103ecc96942db9f931e26677a35d4"
format_stamp: "Formatted at 2026-09-29 15:56:20 on dist-test-slave-r0ck"
server_key: "a329acf225ff29ccae763c66d479ce95"
server_key_iv: "32466b9106649e41fdb29c0f7bc1d327"
server_key_version: "encryptionkey@0"
I20260929 15:56:20.760584 19485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:56:20.781984 19485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:56:20.782904 19485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:56:20.783195 19485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:56:20.784185 19485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:56:20.785844 19485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:56:20.785979 19485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:20.786099 19485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:56:20.786185 19485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:20.817207 19485 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43565
I20260929 15:56:20.817288 19611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43565 every 8 connection(s)
I20260929 15:56:20.818564 19485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:56:20.820305 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 19485
I20260929 15:56:20.820475 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:56:20.820931 388 external_mini_cluster.cc:1428] Setting key 890386d80fd503e6845c164cfe53e4bf
I20260929 15:56:20.831447 19353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:20.821190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54935 (local address 127.0.97.62:46357)
0929 15:56:20.821441 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:20.821445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:20.823202 (+ 1757us) server_negotiation.cc:407] Connection header received
0929 15:56:20.825150 (+ 1948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:20.825155 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:20.825250 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:20.825387 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:20.826410 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:20.827140 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:20.828676 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:20.829086 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:20.829238 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:20.830600 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:20.830605 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:20.830608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:20.830627 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:20.830657 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:20.830669 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:20.830764 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:20.831005 (+ 241us) server_negotiation.cc:299] Negotiation successful
0929 15:56:20.831108 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 15:56:20.832453 19614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:20.821891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46357 (local address 127.0.97.3:54935)
0929 15:56:20.823070 (+ 1179us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:20.823104 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:20.824899 (+ 1795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:20.825613 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:20.825621 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:20.825705 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:56:20.826242 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:20.826257 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:20.827295 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:20.827300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:20.828532 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:20.828544 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:20.829336 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:20.829367 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:20.829615 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:20.830277 (+ 662us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:56:20.830279 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:56:20.830316 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:56:20.830400 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:56:20.830765 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:56:20.830778 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:56:20.830789 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 15:56:20.830979 (+ 190us) client_negotiation.cc:240] Negotiation successful
0929 15:56:20.831375 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":427,"threads_started":1}
I20260929 15:56:20.833909 19612 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46357
I20260929 15:56:20.834223 19612 heartbeater.cc:464] Registering TS with master...
I20260929 15:56:20.835103 19612 heartbeater.cc:511] Master 127.0.97.62:46357 requested a full tablet report, sending...
I20260929 15:56:20.836553 19165 ts_manager.cc:194] Registered new tserver with Master: 0cc103ecc96942db9f931e26677a35d4 (127.0.97.3:43565)
I20260929 15:56:20.837337 19165 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:54935
I20260929 15:56:20.848794 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:56:20.859206 19353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:20.853060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48466 (local address 127.0.97.62:46357)
0929 15:56:20.853274 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:20.853279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:20.853725 (+ 446us) server_negotiation.cc:407] Connection header received
0929 15:56:20.853860 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:20.853863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:20.853949 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:20.854049 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:20.854621 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:20.855328 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:20.856785 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:20.857192 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:20.857313 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:20.858506 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:20.858511 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:20.858515 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:20.858533 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:20.858565 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:20.858578 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:20.858696 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:20.858916 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 15:56:20.858964 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 15:56:20.863112 19165 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48466:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:56:20.870548 19165 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:56:20.919422 19351 heartbeater.cc:503] Master 127.0.97.62:46357 was elected leader, sending a full tablet report...
I20260929 15:56:20.919988 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:20.884925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43565 (local address 127.0.0.1:49530)
0929 15:56:20.912696 (+ 27771us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:20.912718 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:20.912841 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:20.913459 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:20.913468 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:20.913499 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:56:20.913911 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:20.913925 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:20.915626 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:20.915634 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:20.917717 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:20.917732 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:20.919024 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:20.919030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:20.919253 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:20.919310 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 15:56:20.919468 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 15:56:20.919654 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27656,"thread_start_us":25914,"threads_started":1}
I20260929 15:56:20.920437 19629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:20.885759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49530 (local address 127.0.97.3:43565)
0929 15:56:20.911940 (+ 26181us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:20.911953 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:20.912827 (+ 874us) server_negotiation.cc:407] Connection header received
0929 15:56:20.912991 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:20.913001 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:20.913216 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:20.913449 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:56:20.914154 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:20.915487 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:20.917931 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:20.918819 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:20.918991 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:20.919267 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:20.919510 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 15:56:20.919679 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25942,"thread_start_us":25743,"threads_started":1}
I20260929 15:56:20.923265 19628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:20.883121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40113 (local address 127.0.0.1:47840)
0929 15:56:20.911131 (+ 28010us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:20.911158 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:20.911280 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:20.913951 (+ 2671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:20.913955 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:20.914002 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:56:20.914523 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:20.914540 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:20.916572 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:20.916577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:20.919481 (+ 2904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:20.919497 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:20.922447 (+ 2950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:20.922455 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:20.922628 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:20.922663 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 15:56:20.922772 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 15:56:20.922914 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27850,"mutex_wait_us":255,"spinlock_wait_cycles":6912,"thread_start_us":29336,"threads_started":1}
I20260929 15:56:20.923693 19627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:20.883450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47840 (local address 127.0.97.1:40113)
0929 15:56:20.912390 (+ 28940us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:20.912401 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:20.912434 (+ 33us) server_negotiation.cc:407] Connection header received
0929 15:56:20.912547 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:20.912554 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:20.913732 (+ 1178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:20.914634 (+ 902us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:56:20.914871 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:20.916404 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:20.919738 (+ 3334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:20.921784 (+ 2046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:20.921957 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:20.922312 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:20.922854 (+ 542us) server_negotiation.cc:299] Negotiation successful
0929 15:56:20.923104 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28776,"thread_start_us":28654,"threads_started":1}
I20260929 15:56:20.924747 19547 tablet_service.cc:1467] Processing CreateTablet for tablet e365956ced184490ad201781f759151e (DEFAULT_TABLE table=test-table [id=acbb4d4fb969454b83f3d194ae0ca16c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:56:20.925369 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:20.882923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43635 (local address 127.0.0.1:46494)
0929 15:56:20.910139 (+ 27216us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:20.910197 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:20.910417 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:20.913949 (+ 3532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:20.913954 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:20.913994 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:56:20.914560 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:20.914578 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:20.916572 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:20.916577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:20.922157 (+ 5580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:20.922174 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:20.924571 (+ 2397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:20.924577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:20.924745 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:20.924784 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 15:56:20.924883 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 15:56:20.925013 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26939,"mutex_wait_us":222,"spinlock_wait_cycles":896,"thread_start_us":26747,"threads_started":1}
I20260929 15:56:20.925329 19626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:20.883113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46494 (local address 127.0.97.2:43635)
0929 15:56:20.913372 (+ 30259us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:20.913382 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:20.913409 (+ 27us) server_negotiation.cc:407] Connection header received
0929 15:56:20.913499 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:20.913508 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:20.913715 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:20.914300 (+ 585us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:56:20.914884 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:20.916404 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:20.922415 (+ 6011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:20.924359 (+ 1944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:20.924520 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:20.924741 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:20.924940 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 15:56:20.925079 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30059,"thread_start_us":29963,"threads_started":1}
I20260929 15:56:20.926623 19547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e365956ced184490ad201781f759151e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:56:20.927734 19286 tablet_service.cc:1467] Processing CreateTablet for tablet e365956ced184490ad201781f759151e (DEFAULT_TABLE table=test-table [id=acbb4d4fb969454b83f3d194ae0ca16c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:56:20.928808 19286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e365956ced184490ad201781f759151e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:56:20.928911 19417 tablet_service.cc:1467] Processing CreateTablet for tablet e365956ced184490ad201781f759151e (DEFAULT_TABLE table=test-table [id=acbb4d4fb969454b83f3d194ae0ca16c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:56:20.930183 19417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e365956ced184490ad201781f759151e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:56:20.939920 19630 tablet_bootstrap.cc:492] T e365956ced184490ad201781f759151e P bac42ab8c24d46bab5406b255cafee3d: Bootstrap starting.
I20260929 15:56:20.941214 19631 tablet_bootstrap.cc:492] T e365956ced184490ad201781f759151e P 0cc103ecc96942db9f931e26677a35d4: Bootstrap starting.
I20260929 15:56:20.943490 19632 tablet_bootstrap.cc:492] T e365956ced184490ad201781f759151e P b91972eceda742b5a205e113ff6f32c4: Bootstrap starting.
I20260929 15:56:20.944655 19631 tablet_bootstrap.cc:654] T e365956ced184490ad201781f759151e P 0cc103ecc96942db9f931e26677a35d4: Neither blocks nor log segments found. Creating new log.
I20260929 15:56:20.945114 19630 tablet_bootstrap.cc:654] T e365956ced184490ad201781f759151e P bac42ab8c24d46bab5406b255cafee3d: Neither blocks nor log segments found. Creating new log.
I20260929 15:56:20.945937 19631 log.cc:824] T e365956ced184490ad201781f759151e P 0cc103ecc96942db9f931e26677a35d4: Log is configured to *not* fsync() on all Append() calls
I20260929 15:56:20.946328 19630 log.cc:824] T e365956ced184490ad201781f759151e P bac42ab8c24d46bab5406b255cafee3d: Log is configured to *not* fsync() on all Append() calls
I20260929 15:56:20.947001 19632 tablet_bootstrap.cc:654] T e365956ced184490ad201781f759151e P b91972eceda742b5a205e113ff6f32c4: Neither blocks nor log segments found. Creating new log.
I20260929 15:56:20.948217 19632 log.cc:824] T e365956ced184490ad201781f759151e P b91972eceda742b5a205e113ff6f32c4: Log is configured to *not* fsync() on all Append() calls
I20260929 15:56:20.949239 19631 tablet_bootstrap.cc:492] T e365956ced184490ad201781f759151e P 0cc103ecc96942db9f931e26677a35d4: No bootstrap required, opened a new log
I20260929 15:56:20.949316 19630 tablet_bootstrap.cc:492] T e365956ced184490ad201781f759151e P bac42ab8c24d46bab5406b255cafee3d: No bootstrap required, opened a new log
I20260929 15:56:20.949594 19631 ts_tablet_manager.cc:1397] T e365956ced184490ad201781f759151e P 0cc103ecc96942db9f931e26677a35d4: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 15:56:20.949604 19630 ts_tablet_manager.cc:1397] T e365956ced184490ad201781f759151e P bac42ab8c24d46bab5406b255cafee3d: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260929 15:56:20.951023 19632 tablet_bootstrap.cc:492] T e365956ced184490ad201781f759151e P b91972eceda742b5a205e113ff6f32c4: No bootstrap required, opened a new log
I20260929 15:56:20.951256 19632 ts_tablet_manager.cc:1397] T e365956ced184490ad201781f759151e P b91972eceda742b5a205e113ff6f32c4: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 15:56:20.957181 19630 raft_consensus.cc:348] T e365956ced184490ad201781f759151e P bac42ab8c24d46bab5406b255cafee3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b91972eceda742b5a205e113ff6f32c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43635 } } peers { permanent_uuid: "bac42ab8c24d46bab5406b255cafee3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40113 } } peers { permanent_uuid: "0cc103ecc96942db9f931e26677a35d4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43565 } }
I20260929 15:56:20.957592 19630 raft_consensus.cc:374] T e365956ced184490ad201781f759151e P bac42ab8c24d46bab5406b255cafee3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:56:20.957695 19630 raft_consensus.cc:729] T e365956ced184490ad201781f759151e P bac42ab8c24d46bab5406b255cafee3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bac42ab8c24d46bab5406b255cafee3d, State: Initialized, Role: FOLLOWER
I20260929 15:56:20.958302 19630 consensus_queue.cc:260] T e365956ced184490ad201781f759151e P bac42ab8c24d46bab5406b255cafee3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b91972eceda742b5a205e113ff6f32c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43635 } } peers { permanent_uuid: "bac42ab8c24d46bab5406b255cafee3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40113 } } peers { permanent_uuid: "0cc103ecc96942db9f931e26677a35d4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43565 } }
I20260929 15:56:20.959950 19630 ts_tablet_manager.cc:1428] T e365956ced184490ad201781f759151e P bac42ab8c24d46bab5406b255cafee3d: Time spent starting tablet: real 0.010s user 0.007s sys 0.002s
I20260929 15:56:20.959908 19631 raft_consensus.cc:348] T e365956ced184490ad201781f759151e P 0cc103ecc96942db9f931e26677a35d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b91972eceda742b5a205e113ff6f32c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43635 } } peers { permanent_uuid: "bac42ab8c24d46bab5406b255cafee3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40113 } } peers { permanent_uuid: "0cc103ecc96942db9f931e26677a35d4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43565 } }
I20260929 15:56:20.960307 19631 raft_consensus.cc:374] T e365956ced184490ad201781f759151e P 0cc103ecc96942db9f931e26677a35d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:56:20.960448 19631 raft_consensus.cc:729] T e365956ced184490ad201781f759151e P 0cc103ecc96942db9f931e26677a35d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cc103ecc96942db9f931e26677a35d4, State: Initialized, Role: FOLLOWER
I20260929 15:56:20.961009 19631 consensus_queue.cc:260] T e365956ced184490ad201781f759151e P 0cc103ecc96942db9f931e26677a35d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b91972eceda742b5a205e113ff6f32c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43635 } } peers { permanent_uuid: "bac42ab8c24d46bab5406b255cafee3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40113 } } peers { permanent_uuid: "0cc103ecc96942db9f931e26677a35d4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43565 } }
I20260929 15:56:20.961519 19632 raft_consensus.cc:348] T e365956ced184490ad201781f759151e P b91972eceda742b5a205e113ff6f32c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b91972eceda742b5a205e113ff6f32c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43635 } } peers { permanent_uuid: "bac42ab8c24d46bab5406b255cafee3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40113 } } peers { permanent_uuid: "0cc103ecc96942db9f931e26677a35d4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43565 } }
I20260929 15:56:20.962002 19632 raft_consensus.cc:374] T e365956ced184490ad201781f759151e P b91972eceda742b5a205e113ff6f32c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:56:20.962138 19632 raft_consensus.cc:729] T e365956ced184490ad201781f759151e P b91972eceda742b5a205e113ff6f32c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b91972eceda742b5a205e113ff6f32c4, State: Initialized, Role: FOLLOWER
I20260929 15:56:20.962211 19612 heartbeater.cc:503] Master 127.0.97.62:46357 was elected leader, sending a full tablet report...
I20260929 15:56:20.962721 19631 ts_tablet_manager.cc:1428] T e365956ced184490ad201781f759151e P 0cc103ecc96942db9f931e26677a35d4: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260929 15:56:20.962844 19632 consensus_queue.cc:260] T e365956ced184490ad201781f759151e P b91972eceda742b5a205e113ff6f32c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b91972eceda742b5a205e113ff6f32c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43635 } } peers { permanent_uuid: "bac42ab8c24d46bab5406b255cafee3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40113 } } peers { permanent_uuid: "0cc103ecc96942db9f931e26677a35d4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43565 } }
I20260929 15:56:20.965291 19482 heartbeater.cc:503] Master 127.0.97.62:46357 was elected leader, sending a full tablet report...
I20260929 15:56:20.966032 19632 ts_tablet_manager.cc:1428] T e365956ced184490ad201781f759151e P b91972eceda742b5a205e113ff6f32c4: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260929 15:56:20.985087 19638 raft_consensus.cc:482] T e365956ced184490ad201781f759151e P b91972eceda742b5a205e113ff6f32c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:56:20.985477 19638 raft_consensus.cc:504] T e365956ced184490ad201781f759151e P b91972eceda742b5a205e113ff6f32c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b91972eceda742b5a205e113ff6f32c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43635 } } peers { permanent_uuid: "bac42ab8c24d46bab5406b255cafee3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40113 } } peers { permanent_uuid: "0cc103ecc96942db9f931e26677a35d4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43565 } }
I20260929 15:56:20.987179 19638 leader_election.cc:288] T e365956ced184490ad201781f759151e P b91972eceda742b5a205e113ff6f32c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bac42ab8c24d46bab5406b255cafee3d (127.0.97.1:40113), 0cc103ecc96942db9f931e26677a35d4 (127.0.97.3:43565)
I20260929 15:56:20.994585 19627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:20.987604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46057 (local address 127.0.97.1:40113)
0929 15:56:20.987839 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:20.987848 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:20.988589 (+ 741us) server_negotiation.cc:407] Connection header received
0929 15:56:20.988819 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:20.988825 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:20.988911 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:20.988980 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:56:20.989726 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:20.990597 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:20.992557 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:20.993613 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:20.993760 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:20.993875 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:20.994204 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 15:56:20.994337 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 15:56:20.994647 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:20.987329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40113 (local address 127.0.97.2:46057)
0929 15:56:20.988479 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:20.988495 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:20.988661 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:20.989026 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:20.989032 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:20.989058 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:56:20.989565 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:20.989578 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:20.990722 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:20.990730 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:20.992364 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:20.992375 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:20.993823 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:20.993829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:20.994034 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:20.994066 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 15:56:20.994167 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 15:56:20.994294 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1004,"mutex_wait_us":144,"thread_start_us":704,"threads_started":1}
I20260929 15:56:20.994743 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:20.987627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43565 (local address 127.0.97.2:39663)
0929 15:56:20.988491 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:20.988505 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:20.988700 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:20.989073 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:20.989077 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:20.989100 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:56:20.989656 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:20.989673 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:20.990787 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:20.990794 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:20.992377 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:20.992388 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:20.994024 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:20.994030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:20.994202 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:20.994229 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 15:56:20.994315 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 15:56:20.994417 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"mutex_wait_us":84,"spinlock_wait_cycles":15104,"thread_start_us":504,"threads_started":1}
I20260929 15:56:20.994743 19629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:20.987650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39663 (local address 127.0.97.3:43565)
0929 15:56:20.987888 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:20.987897 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:20.988631 (+ 734us) server_negotiation.cc:407] Connection header received
0929 15:56:20.988870 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:20.988874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:20.988931 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:20.989046 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:56:20.989855 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:20.990629 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:20.992779 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:20.993846 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:20.993959 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:20.994058 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:20.994376 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 15:56:20.994498 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 15:56:20.995662 19567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e365956ced184490ad201781f759151e" candidate_uuid: "b91972eceda742b5a205e113ff6f32c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cc103ecc96942db9f931e26677a35d4" is_pre_election: true
I20260929 15:56:20.995971 19306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e365956ced184490ad201781f759151e" candidate_uuid: "b91972eceda742b5a205e113ff6f32c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bac42ab8c24d46bab5406b255cafee3d" is_pre_election: true
I20260929 15:56:20.996163 19567 raft_consensus.cc:2454] T e365956ced184490ad201781f759151e P 0cc103ecc96942db9f931e26677a35d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b91972eceda742b5a205e113ff6f32c4 in term 0.
I20260929 15:56:20.996469 19306 raft_consensus.cc:2454] T e365956ced184490ad201781f759151e P bac42ab8c24d46bab5406b255cafee3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b91972eceda742b5a205e113ff6f32c4 in term 0.
I20260929 15:56:20.996898 19370 leader_election.cc:302] T e365956ced184490ad201781f759151e P b91972eceda742b5a205e113ff6f32c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cc103ecc96942db9f931e26677a35d4, b91972eceda742b5a205e113ff6f32c4; no voters:
I20260929 15:56:20.997313 19638 raft_consensus.cc:2781] T e365956ced184490ad201781f759151e P b91972eceda742b5a205e113ff6f32c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:56:20.997488 19638 raft_consensus.cc:482] T e365956ced184490ad201781f759151e P b91972eceda742b5a205e113ff6f32c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:56:20.997601 19638 raft_consensus.cc:3037] T e365956ced184490ad201781f759151e P b91972eceda742b5a205e113ff6f32c4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:56:20.999662 19638 raft_consensus.cc:504] T e365956ced184490ad201781f759151e P b91972eceda742b5a205e113ff6f32c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b91972eceda742b5a205e113ff6f32c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43635 } } peers { permanent_uuid: "bac42ab8c24d46bab5406b255cafee3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40113 } } peers { permanent_uuid: "0cc103ecc96942db9f931e26677a35d4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43565 } }
I20260929 15:56:21.000212 19638 leader_election.cc:288] T e365956ced184490ad201781f759151e P b91972eceda742b5a205e113ff6f32c4 [CANDIDATE]: Term 1 election: Requested vote from peers bac42ab8c24d46bab5406b255cafee3d (127.0.97.1:40113), 0cc103ecc96942db9f931e26677a35d4 (127.0.97.3:43565)
I20260929 15:56:21.000646 19306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e365956ced184490ad201781f759151e" candidate_uuid: "b91972eceda742b5a205e113ff6f32c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bac42ab8c24d46bab5406b255cafee3d"
I20260929 15:56:21.000697 19567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e365956ced184490ad201781f759151e" candidate_uuid: "b91972eceda742b5a205e113ff6f32c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cc103ecc96942db9f931e26677a35d4"
I20260929 15:56:21.000836 19306 raft_consensus.cc:3037] T e365956ced184490ad201781f759151e P bac42ab8c24d46bab5406b255cafee3d [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:56:21.000911 19567 raft_consensus.cc:3037] T e365956ced184490ad201781f759151e P 0cc103ecc96942db9f931e26677a35d4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:56:21.003058 19306 raft_consensus.cc:2454] T e365956ced184490ad201781f759151e P bac42ab8c24d46bab5406b255cafee3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b91972eceda742b5a205e113ff6f32c4 in term 1.
I20260929 15:56:21.003496 19371 leader_election.cc:302] T e365956ced184490ad201781f759151e P b91972eceda742b5a205e113ff6f32c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b91972eceda742b5a205e113ff6f32c4, bac42ab8c24d46bab5406b255cafee3d; no voters:
I20260929 15:56:21.003782 19567 raft_consensus.cc:2454] T e365956ced184490ad201781f759151e P 0cc103ecc96942db9f931e26677a35d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b91972eceda742b5a205e113ff6f32c4 in term 1.
I20260929 15:56:21.003847 19638 raft_consensus.cc:2781] T e365956ced184490ad201781f759151e P b91972eceda742b5a205e113ff6f32c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:56:21.004374 19638 raft_consensus.cc:686] T e365956ced184490ad201781f759151e P b91972eceda742b5a205e113ff6f32c4 [term 1 LEADER]: Becoming Leader. State: Replica: b91972eceda742b5a205e113ff6f32c4, State: Running, Role: LEADER
I20260929 15:56:21.004977 19638 consensus_queue.cc:237] T e365956ced184490ad201781f759151e P b91972eceda742b5a205e113ff6f32c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b91972eceda742b5a205e113ff6f32c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43635 } } peers { permanent_uuid: "bac42ab8c24d46bab5406b255cafee3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40113 } } peers { permanent_uuid: "0cc103ecc96942db9f931e26677a35d4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43565 } }
I20260929 15:56:21.010658 19164 catalog_manager.cc:5495] T e365956ced184490ad201781f759151e P b91972eceda742b5a205e113ff6f32c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to b91972eceda742b5a205e113ff6f32c4 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "b91972eceda742b5a205e113ff6f32c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b91972eceda742b5a205e113ff6f32c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bac42ab8c24d46bab5406b255cafee3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cc103ecc96942db9f931e26677a35d4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43565 } health_report { overall_health: UNKNOWN } } }
I20260929 15:56:21.050809 19626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:21.045538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46502 (local address 127.0.97.2:43635)
0929 15:56:21.045727 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:21.045732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:21.045749 (+ 17us) server_negotiation.cc:407] Connection header received
0929 15:56:21.045882 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:21.045886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:21.045972 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:21.046092 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:56:21.046462 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:21.047189 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:21.048526 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:21.048933 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:21.049032 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:21.049664 (+ 632us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:56:21.050159 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:56:21.050289 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:21.050449 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 15:56:21.050514 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 15:56:21.058631 19306 raft_consensus.cc:1261] T e365956ced184490ad201781f759151e P bac42ab8c24d46bab5406b255cafee3d [term 1 FOLLOWER]: Refusing update from remote peer b91972eceda742b5a205e113ff6f32c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:56:21.058674 19567 raft_consensus.cc:1261] T e365956ced184490ad201781f759151e P 0cc103ecc96942db9f931e26677a35d4 [term 1 FOLLOWER]: Refusing update from remote peer b91972eceda742b5a205e113ff6f32c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:56:21.059572 19641 consensus_queue.cc:1035] T e365956ced184490ad201781f759151e P b91972eceda742b5a205e113ff6f32c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cc103ecc96942db9f931e26677a35d4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:56:21.059888 19638 consensus_queue.cc:1035] T e365956ced184490ad201781f759151e P b91972eceda742b5a205e113ff6f32c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bac42ab8c24d46bab5406b255cafee3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:56:21.070518 19644 mvcc.cc:205] Tried to move back new op lower bound from 7334696472806952960 to 7334696472602857472. Current Snapshot: MvccSnapshot[applied={T|T < 7334696472806952960}]
W20260929 15:56:21.070709 19613 tablet.cc:2369] T e365956ced184490ad201781f759151e P 0cc103ecc96942db9f931e26677a35d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:56:21.084395 19164 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:56:21.084704 19164 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:56:21.088311 19164 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 07d068e6272248a38d07bdf311b1bf1e: Sending DeleteTablet for 3 replicas of tablet e365956ced184490ad201781f759151e
I20260929 15:56:21.089429 19417 tablet_service.cc:1514] Processing DeleteTablet for tablet e365956ced184490ad201781f759151e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:56:21 UTC) from {username='slave'} at 127.0.0.1:46494
I20260929 15:56:21.089583 19286 tablet_service.cc:1514] Processing DeleteTablet for tablet e365956ced184490ad201781f759151e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:56:21 UTC) from {username='slave'} at 127.0.0.1:47840
I20260929 15:56:21.089634 19547 tablet_service.cc:1514] Processing DeleteTablet for tablet e365956ced184490ad201781f759151e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:56:21 UTC) from {username='slave'} at 127.0.0.1:49530
I20260929 15:56:21.090688 19653 tablet_replica.cc:330] T e365956ced184490ad201781f759151e P b91972eceda742b5a205e113ff6f32c4: stopping tablet replica
I20260929 15:56:21.091140 19653 raft_consensus.cc:2229] T e365956ced184490ad201781f759151e P b91972eceda742b5a205e113ff6f32c4 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:56:21.091238 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 19224
I20260929 15:56:21.091887 19653 raft_consensus.cc:2258] T e365956ced184490ad201781f759151e P b91972eceda742b5a205e113ff6f32c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:56:21.092589 19654 tablet_replica.cc:330] T e365956ced184490ad201781f759151e P 0cc103ecc96942db9f931e26677a35d4: stopping tablet replica
I20260929 15:56:21.092926 19654 raft_consensus.cc:2229] T e365956ced184490ad201781f759151e P 0cc103ecc96942db9f931e26677a35d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:56:21.093408 19654 raft_consensus.cc:2258] T e365956ced184490ad201781f759151e P 0cc103ecc96942db9f931e26677a35d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:56:21.094146 19655 tablet_replica.cc:330] T e365956ced184490ad201781f759151e P bac42ab8c24d46bab5406b255cafee3d: stopping tablet replica
I20260929 15:56:21.094519 19655 raft_consensus.cc:2229] T e365956ced184490ad201781f759151e P bac42ab8c24d46bab5406b255cafee3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:56:21.095372 19655 raft_consensus.cc:2258] T e365956ced184490ad201781f759151e P bac42ab8c24d46bab5406b255cafee3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:56:21.096361 19653 ts_tablet_manager.cc:1905] T e365956ced184490ad201781f759151e P b91972eceda742b5a205e113ff6f32c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:56:21.097620 19654 ts_tablet_manager.cc:1905] T e365956ced184490ad201781f759151e P 0cc103ecc96942db9f931e26677a35d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:56:21.103919 19653 ts_tablet_manager.cc:1918] T e365956ced184490ad201781f759151e P b91972eceda742b5a205e113ff6f32c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:56:21.103979 19627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:21.096030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47852 (local address 127.0.97.1:40113)
0929 15:56:21.096193 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:21.096199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:21.096217 (+ 18us) server_negotiation.cc:407] Connection header received
0929 15:56:21.096275 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:21.096280 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:21.096377 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:21.096483 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:21.097076 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:21.098178 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:21.100089 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:21.100674 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:21.100788 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:21.102329 (+ 1541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:21.102339 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:21.102358 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:21.103086 (+ 728us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:21.103142 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:21.103165 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:21.103243 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:21.103468 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 15:56:21.103581 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 15:56:21.104223 19653 log.cc:1196] T e365956ced184490ad201781f759151e P b91972eceda742b5a205e113ff6f32c4: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-1/wal/wals/e365956ced184490ad201781f759151e
I20260929 15:56:21.104772 19653 ts_tablet_manager.cc:1939] T e365956ced184490ad201781f759151e P b91972eceda742b5a205e113ff6f32c4: Deleting consensus metadata
I20260929 15:56:21.105439 19346 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:56:21.106845 19654 ts_tablet_manager.cc:1918] T e365956ced184490ad201781f759151e P 0cc103ecc96942db9f931e26677a35d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:56:21.106868 19151 catalog_manager.cc:4841] TS b91972eceda742b5a205e113ff6f32c4 (127.0.97.2:43635): tablet e365956ced184490ad201781f759151e (table test-table [id=acbb4d4fb969454b83f3d194ae0ca16c]) successfully deleted
I20260929 15:56:21.107121 19654 log.cc:1196] T e365956ced184490ad201781f759151e P 0cc103ecc96942db9f931e26677a35d4: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697075657932-388-0/minicluster-data/ts-2/wal/wals/e365956ced184490ad201781f759151e
I20260929 15:56:21.107694 19654 ts_tablet_manager.cc:1939] T e365956ced184490ad201781f759151e P 0cc103ecc96942db9f931e26677a35d4: Deleting consensus metadata
I20260929 15:56:21.109441 19149 catalog_manager.cc:4841] TS 0cc103ecc96942db9f931e26677a35d4 (127.0.97.3:43565): tablet e365956ced184490ad201781f759151e (table test-table [id=acbb4d4fb969454b83f3d194ae0ca16c]) successfully deleted
I20260929 15:56:21.106443 19655 ts_tablet_manager.cc:1905] T e365956ced184490ad201781f759151e P bac42ab8c24d46bab5406b255cafee3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:56:21.230063 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 19224
I20260929 15:56:21.247503 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 19355
W20260929 15:56:21.247591 19150 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:40113 (error 108)
I20260929 15:56:21.249444 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:21.248959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40113 (local address 127.0.0.1:47856)
0929 15:56:21.249106 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:21.249235 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 15:56:21.249632 19150 catalog_manager.cc:4584] TS bac42ab8c24d46bab5406b255cafee3d (127.0.97.1:40113): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e365956ced184490ad201781f759151e: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40113: connect: Connection refused (error 111)
I20260929 15:56:21.256320 19626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:21.248637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46516 (local address 127.0.97.2:43635)
0929 15:56:21.248833 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:21.248839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:21.248858 (+ 19us) server_negotiation.cc:407] Connection header received
0929 15:56:21.249240 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:21.249245 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:21.249395 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:21.249511 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:21.250383 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:21.251152 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:21.253006 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:21.253531 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:21.253630 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:21.254751 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:21.254760 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:21.254773 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:21.255560 (+ 787us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:21.255604 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:21.255623 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:21.255710 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:21.255867 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 15:56:21.255943 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 15:56:21.257012 19477 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:56:21.303334 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:21.302780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40113 (local address 127.0.0.1:47870)
0929 15:56:21.303026 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:21.303135 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260929 15:56:21.304356 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:21.303975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40113 (local address 127.0.0.1:47878)
0929 15:56:21.304134 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:21.304230 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 15:56:21.364802 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:21.364230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40113 (local address 127.0.0.1:47892)
0929 15:56:21.364457 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:21.364567 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 15:56:21.365847 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:21.365485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40113 (local address 127.0.0.1:47896)
0929 15:56:21.365616 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:21.365714 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 15:56:21.376749 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 19355
I20260929 15:56:21.395121 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 19485
I20260929 15:56:21.401636 19629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:21.395874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49540 (local address 127.0.97.3:43565)
0929 15:56:21.396056 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:21.396060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:21.396078 (+ 18us) server_negotiation.cc:407] Connection header received
0929 15:56:21.396217 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:21.396221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:21.396297 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:21.396383 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:21.396720 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:21.397536 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:21.398810 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:21.399182 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:21.399271 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:21.400284 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:21.400292 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:21.400306 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:21.400841 (+ 535us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:21.400895 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:21.400913 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:21.400998 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:21.401130 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 15:56:21.401210 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 15:56:21.402303 19607 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:56:21.446357 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:21.445858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40113 (local address 127.0.0.1:47898)
0929 15:56:21.446074 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:21.446180 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 15:56:21.447585 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:21.447168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40113 (local address 127.0.0.1:47910)
0929 15:56:21.447380 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:21.447461 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260929 15:56:21.493970 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 19485
I20260929 15:56:21.511368 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 19134
I20260929 15:56:21.512405 19195 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:56:21.621302 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:21.620793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40113 (local address 127.0.0.1:47914)
0929 15:56:21.621027 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:21.621124 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 15:56:21.621840 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 19134
2026-09-29T15:56:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2797 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:56:21 dist-test-slave-r0ck krb5kdc[19664](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:56:21 dist-test-slave-r0ck krb5kdc[19664](info): set up 2 sockets
Sep 29 15:56:21 dist-test-slave-r0ck krb5kdc[19664](info): commencing operation
krb5kdc: starting...
W20260929 15:56:22.733796 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:56:22 dist-test-slave-r0ck krb5kdc[19664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:56:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:56:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:56:23.025733 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33557
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33557
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:56:23.370302 19677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:56:23.370666 19677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:56:23.370855 19677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:56:23.379186 19677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:56:23.379333 19677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:56:23.379433 19677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:56:23.379550 19677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:56:23.379736 19677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:56:23.391414 19677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:56:23.392755 19677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:56:23.394229 19677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:56:23.406553 19682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:23.406582 19683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:23.406804 19685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:56:23.407549 19677 server_base.cc:1034] running on GCE node
I20260929 15:56:23.408830 19677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:56:23.410966 19677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:56:23.412415 19677 hybrid_clock.cc:648] HybridClock initialized: now 1790697383412334 us; error 58 us; skew 500 ppm
I20260929 15:56:23.412855 19677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:56:23 dist-test-slave-r0ck krb5kdc[19664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:56:23.420812 19677 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:56:23.422509 19677 webserver.cc:460] Webserver started at http://127.0.97.62:33885/ using document root <none> and password file <none>
I20260929 15:56:23.423233 19677 fs_manager.cc:359] Metadata directory not provided
I20260929 15:56:23.423380 19677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:56:23.423689 19677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:56:23.426942 19677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "53d544cfe74841b68902563da5810801"
format_stamp: "Formatted at 2026-09-29 15:56:23 on dist-test-slave-r0ck"
server_key: "1bbe06ec3672f7b646f2c776fa7f1f68"
server_key_iv: "81834b0f8a4e22bb5f7d0dec63bb5e27"
server_key_version: "encryptionkey@0"
I20260929 15:56:23.427688 19677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "53d544cfe74841b68902563da5810801"
format_stamp: "Formatted at 2026-09-29 15:56:23 on dist-test-slave-r0ck"
server_key: "1bbe06ec3672f7b646f2c776fa7f1f68"
server_key_iv: "81834b0f8a4e22bb5f7d0dec63bb5e27"
server_key_version: "encryptionkey@0"
I20260929 15:56:23.435631 19677 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.002s sys 0.007s
I20260929 15:56:23.439607 19692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:23.441175 19677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 15:56:23.441468 19677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "53d544cfe74841b68902563da5810801"
format_stamp: "Formatted at 2026-09-29 15:56:23 on dist-test-slave-r0ck"
server_key: "1bbe06ec3672f7b646f2c776fa7f1f68"
server_key_iv: "81834b0f8a4e22bb5f7d0dec63bb5e27"
server_key_version: "encryptionkey@0"
I20260929 15:56:23.441691 19677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:56:23.458077 19677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:56:23.462499 19677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:56:23.462786 19677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:56:23.482575 19677 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33557
I20260929 15:56:23.482599 19743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33557 every 8 connection(s)
I20260929 15:56:23.483901 19677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:56:23.484997 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 19677
I20260929 15:56:23.485201 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:56:23.485706 388 external_mini_cluster.cc:1428] Setting key 31942cc61c58dd9c6cd8ed5cd0553542
I20260929 15:56:23.488236 19744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 15:56:23 dist-test-slave-r0ck krb5kdc[19664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:56:23.502848 19744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53d544cfe74841b68902563da5810801: Bootstrap starting.
I20260929 15:56:23.507771 19744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53d544cfe74841b68902563da5810801: Neither blocks nor log segments found. Creating new log.
I20260929 15:56:23.509289 19744 log.cc:824] T 00000000000000000000000000000000 P 53d544cfe74841b68902563da5810801: Log is configured to *not* fsync() on all Append() calls
I20260929 15:56:23.512145 19747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:23.487421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57202 (local address 127.0.97.62:33557)
0929 15:56:23.488181 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:23.488192 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:23.488257 (+ 65us) server_negotiation.cc:407] Connection header received
0929 15:56:23.489945 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:23.489966 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:23.490313 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:23.490625 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:23.492444 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:23.493838 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:23.495771 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:23.496439 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:23.496571 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:23.502338 (+ 5767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:23.502352 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:23.502378 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:56:23.502443 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:23.506807 (+ 4364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:23.507742 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:23.507759 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:23.507769 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:23.507982 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:23.508392 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:23.508396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:23.508399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:23.509182 (+ 783us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:23.509521 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:23.510167 (+ 646us) server_negotiation.cc:299] Negotiation successful
0929 15:56:23.510658 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":362,"threads_started":1}
I20260929 15:56:23.513758 19744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53d544cfe74841b68902563da5810801: No bootstrap required, opened a new log
I20260929 15:56:23.523377 19744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53d544cfe74841b68902563da5810801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d544cfe74841b68902563da5810801" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33557 } }
I20260929 15:56:23.524012 19744 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53d544cfe74841b68902563da5810801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:56:23.524132 19744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53d544cfe74841b68902563da5810801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53d544cfe74841b68902563da5810801, State: Initialized, Role: FOLLOWER
I20260929 15:56:23.524780 19744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53d544cfe74841b68902563da5810801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d544cfe74841b68902563da5810801" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33557 } }
I20260929 15:56:23.525019 19744 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53d544cfe74841b68902563da5810801 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:56:23.525189 19744 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53d544cfe74841b68902563da5810801 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:56:23.525475 19744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53d544cfe74841b68902563da5810801 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:56:23.527961 19744 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53d544cfe74841b68902563da5810801 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d544cfe74841b68902563da5810801" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33557 } }
I20260929 15:56:23.528697 19744 leader_election.cc:302] T 00000000000000000000000000000000 P 53d544cfe74841b68902563da5810801 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53d544cfe74841b68902563da5810801; no voters:
I20260929 15:56:23.529464 19744 leader_election.cc:288] T 00000000000000000000000000000000 P 53d544cfe74841b68902563da5810801 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:56:23.529655 19749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53d544cfe74841b68902563da5810801 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:56:23.531095 19749 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53d544cfe74841b68902563da5810801 [term 1 LEADER]: Becoming Leader. State: Replica: 53d544cfe74841b68902563da5810801, State: Running, Role: LEADER
I20260929 15:56:23.531639 19749 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53d544cfe74841b68902563da5810801 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d544cfe74841b68902563da5810801" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33557 } }
I20260929 15:56:23.532202 19744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53d544cfe74841b68902563da5810801 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:56:23.536211 19751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53d544cfe74841b68902563da5810801 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53d544cfe74841b68902563da5810801. Latest consensus state: current_term: 1 leader_uuid: "53d544cfe74841b68902563da5810801" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d544cfe74841b68902563da5810801" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33557 } } }
I20260929 15:56:23.537142 19751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53d544cfe74841b68902563da5810801 [sys.catalog]: This master's current role is: LEADER
I20260929 15:56:23.536856 19750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53d544cfe74841b68902563da5810801 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53d544cfe74841b68902563da5810801" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d544cfe74841b68902563da5810801" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33557 } } }
I20260929 15:56:23.537377 19750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53d544cfe74841b68902563da5810801 [sys.catalog]: This master's current role is: LEADER
I20260929 15:56:23.541006 19756 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:56:23.546826 19756 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:56:23.556551 19756 catalog_manager.cc:1347] Generated new cluster ID: 5e9a5cbdc25f4cdfb9a8170acf34bbbd
I20260929 15:56:23.556664 19756 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:56:23.574126 19756 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:56:23.575949 19756 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:56:23.593215 19756 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53d544cfe74841b68902563da5810801: Generated new TSK 0
I20260929 15:56:23.594070 19756 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 15:56:23.672914 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33557
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:56:24.019811 19770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:56:24.020103 19770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:56:24.020257 19770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:56:24.027645 19770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:56:24.027752 19770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:56:24.027915 19770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:56:24.038165 19770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:56:24.039569 19770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:56:24.040941 19770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:56:24.054693 19775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:24.054620 19778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:24.054618 19776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:56:24.055460 19770 server_base.cc:1034] running on GCE node
I20260929 15:56:24.056038 19770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:56:24.057036 19770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:56:24.058441 19770 hybrid_clock.cc:648] HybridClock initialized: now 1790697384058311 us; error 50 us; skew 500 ppm
I20260929 15:56:24.059098 19770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:56:24 dist-test-slave-r0ck krb5kdc[19664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:56:24.063124 19770 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:56:24.065009 19770 webserver.cc:460] Webserver started at http://127.0.97.1:41589/ using document root <none> and password file <none>
I20260929 15:56:24.065901 19770 fs_manager.cc:359] Metadata directory not provided
I20260929 15:56:24.066013 19770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:56:24.066304 19770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:56:24.069900 19770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-0/data/instance:
uuid: "2706ab21de13409bb2b2488321cddc28"
format_stamp: "Formatted at 2026-09-29 15:56:24 on dist-test-slave-r0ck"
server_key: "4c174761e678fed8f578de412c6e58dc"
server_key_iv: "c5e02c1a4acbf081f5025e9cbd345675"
server_key_version: "encryptionkey@0"
I20260929 15:56:24.070761 19770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "2706ab21de13409bb2b2488321cddc28"
format_stamp: "Formatted at 2026-09-29 15:56:24 on dist-test-slave-r0ck"
server_key: "4c174761e678fed8f578de412c6e58dc"
server_key_iv: "c5e02c1a4acbf081f5025e9cbd345675"
server_key_version: "encryptionkey@0"
I20260929 15:56:24.076753 19770 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 15:56:24.080773 19785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:24.082526 19770 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 15:56:24.082720 19770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-0/wal
uuid: "2706ab21de13409bb2b2488321cddc28"
format_stamp: "Formatted at 2026-09-29 15:56:24 on dist-test-slave-r0ck"
server_key: "4c174761e678fed8f578de412c6e58dc"
server_key_iv: "c5e02c1a4acbf081f5025e9cbd345675"
server_key_version: "encryptionkey@0"
I20260929 15:56:24.082965 19770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:56:24.119411 19770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:56:24.123857 19770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:56:24.124176 19770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:56:24.125275 19770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:56:24.126677 19770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:56:24.126751 19770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:24.126881 19770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:56:24.126952 19770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:24.164520 19770 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34357
I20260929 15:56:24.164618 19897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34357 every 8 connection(s)
I20260929 15:56:24.165879 19770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:56:24.172098 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 19770
I20260929 15:56:24.172343 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:56:24.172855 388 external_mini_cluster.cc:1428] Setting key 663d6d4bcc52d4f2df52f46b064472f6
Sep 29 15:56:24 dist-test-slave-r0ck krb5kdc[19664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:56:24.189381 19900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:24.168597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49745 (local address 127.0.97.62:33557)
0929 15:56:24.169559 (+ 962us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:24.169564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:24.170016 (+ 452us) server_negotiation.cc:407] Connection header received
0929 15:56:24.171819 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:24.171825 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:24.171921 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:24.172040 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:24.173689 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:24.174589 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:24.177461 (+ 2872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:24.177851 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:24.177958 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:24.182183 (+ 4225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:24.182189 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:24.182201 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:56:24.182234 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:24.185262 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:24.186328 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:24.186334 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:24.186338 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:24.186478 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:24.186952 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:24.186961 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:24.186963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:24.187379 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:24.187548 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:24.188565 (+ 1017us) server_negotiation.cc:299] Negotiation successful
0929 15:56:24.188860 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":396,"threads_started":1}
I20260929 15:56:24.190330 19901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:24.168952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33557 (local address 127.0.97.1:49745)
0929 15:56:24.169808 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:24.169853 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:24.171538 (+ 1685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:24.172296 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:24.172308 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:24.172875 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:56:24.173524 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:24.173541 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:24.175193 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:24.175198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:24.177260 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:24.177278 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:24.178110 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:24.178120 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:24.178430 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:24.179156 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:56:24.179195 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:56:24.181954 (+ 2759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:56:24.185547 (+ 3593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:56:24.185560 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:56:24.185590 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:56:24.186162 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:56:24.186602 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:56:24.186607 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:56:24.186611 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:56:24.186821 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:56:24.187516 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:56:24.187525 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:56:24.187961 (+ 436us) client_negotiation.cc:769] Sending connection context
0929 15:56:24.188358 (+ 397us) client_negotiation.cc:240] Negotiation successful
0929 15:56:24.188893 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":403,"threads_started":1}
I20260929 15:56:24.192292 19898 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33557
I20260929 15:56:24.192636 19898 heartbeater.cc:464] Registering TS with master...
I20260929 15:56:24.193475 19898 heartbeater.cc:511] Master 127.0.97.62:33557 requested a full tablet report, sending...
I20260929 15:56:24.195581 19709 ts_manager.cc:194] Registered new tserver with Master: 2706ab21de13409bb2b2488321cddc28 (127.0.97.1:34357)
I20260929 15:56:24.197031 19709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49745
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 15:56:24.234395 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33557
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:56:24.576190 19904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:56:24.576464 19904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:56:24.576558 19904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:56:24.584278 19904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:56:24.584604 19904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:56:24.584785 19904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:56:24.596727 19904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:56:24.598009 19904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:56:24.599242 19904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:56:24.613610 19909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:24.613649 19910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:24.613559 19912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:56:24.614843 19904 server_base.cc:1034] running on GCE node
I20260929 15:56:24.615425 19904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:56:24.616380 19904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:56:24.617586 19904 hybrid_clock.cc:648] HybridClock initialized: now 1790697384617549 us; error 32 us; skew 500 ppm
I20260929 15:56:24.617906 19904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:56:24 dist-test-slave-r0ck krb5kdc[19664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:56:24.621188 19904 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:56:24.622788 19904 webserver.cc:460] Webserver started at http://127.0.97.2:40485/ using document root <none> and password file <none>
I20260929 15:56:24.623459 19904 fs_manager.cc:359] Metadata directory not provided
I20260929 15:56:24.623558 19904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:56:24.623860 19904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:56:24.626896 19904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-1/data/instance:
uuid: "0abe277da2c944ef9c91007ec94d9bf4"
format_stamp: "Formatted at 2026-09-29 15:56:24 on dist-test-slave-r0ck"
server_key: "3fa7165e894380ee77659f2d767cba15"
server_key_iv: "578cde0c129f40baf33d5aca1ab7b2c2"
server_key_version: "encryptionkey@0"
I20260929 15:56:24.627661 19904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0abe277da2c944ef9c91007ec94d9bf4"
format_stamp: "Formatted at 2026-09-29 15:56:24 on dist-test-slave-r0ck"
server_key: "3fa7165e894380ee77659f2d767cba15"
server_key_iv: "578cde0c129f40baf33d5aca1ab7b2c2"
server_key_version: "encryptionkey@0"
I20260929 15:56:24.632845 19904 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 15:56:24.636404 19919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:24.637830 19904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:56:24.638006 19904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-1/wal
uuid: "0abe277da2c944ef9c91007ec94d9bf4"
format_stamp: "Formatted at 2026-09-29 15:56:24 on dist-test-slave-r0ck"
server_key: "3fa7165e894380ee77659f2d767cba15"
server_key_iv: "578cde0c129f40baf33d5aca1ab7b2c2"
server_key_version: "encryptionkey@0"
I20260929 15:56:24.638208 19904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:56:24.698513 19904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:56:24.703086 19904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:56:24.703385 19904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:56:24.704362 19904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:56:24.705893 19904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:56:24.705996 19904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:24.706147 19904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:56:24.706221 19904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:24.742671 19904 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35101
I20260929 15:56:24.742766 20031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35101 every 8 connection(s)
I20260929 15:56:24.744016 19904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:56:24.744777 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 19904
I20260929 15:56:24.744994 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:56:24.745819 388 external_mini_cluster.cc:1428] Setting key 158d3c74a369aac45d4fb5075c56903f
Sep 29 15:56:24 dist-test-slave-r0ck krb5kdc[19664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:56:24.769212 20034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:24.748065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56549 (local address 127.0.97.62:33557)
0929 15:56:24.748926 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:24.748935 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:24.750105 (+ 1170us) server_negotiation.cc:407] Connection header received
0929 15:56:24.752361 (+ 2256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:24.752369 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:24.752463 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:24.752601 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:24.754497 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:24.755679 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:24.757317 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:24.757782 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:24.757933 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:24.761648 (+ 3715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:24.761657 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:24.761665 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:56:24.761711 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:24.765694 (+ 3983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:24.766493 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:24.766501 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:24.766508 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:24.766670 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:24.767143 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:24.767151 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:24.767157 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:24.767568 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:24.767745 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:24.768461 (+ 716us) server_negotiation.cc:299] Negotiation successful
0929 15:56:24.768713 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":571,"threads_started":1}
I20260929 15:56:24.770162 20035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:24.748747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33557 (local address 127.0.97.2:56549)
0929 15:56:24.749882 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:24.749937 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:24.752039 (+ 2102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:24.752921 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:24.752933 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:24.753636 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:56:24.754279 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:24.754299 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:24.755861 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:24.755866 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:24.757137 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:24.757150 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:24.758029 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:24.758036 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:24.758275 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:24.759003 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:56:24.759027 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:56:24.761320 (+ 2293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:56:24.765880 (+ 4560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:56:24.765887 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:56:24.765901 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:56:24.766341 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:56:24.766789 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:56:24.766793 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:56:24.766795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:56:24.767007 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:56:24.767701 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:56:24.767707 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:56:24.768052 (+ 345us) client_negotiation.cc:769] Sending connection context
0929 15:56:24.768292 (+ 240us) client_negotiation.cc:240] Negotiation successful
0929 15:56:24.768909 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"thread_start_us":497,"threads_started":1}
I20260929 15:56:24.771685 20032 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33557
I20260929 15:56:24.772002 20032 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 15:56:24.772765 20032 heartbeater.cc:511] Master 127.0.97.62:33557 requested a full tablet report, sending...
I20260929 15:56:24.774330 19709 ts_manager.cc:194] Registered new tserver with Master: 0abe277da2c944ef9c91007ec94d9bf4 (127.0.97.2:35101)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 15:56:24.775273 19709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:56549
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 15:56:24.805038 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33557
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:56:25.137740 20038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:56:25.137997 20038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:56:25.138089 20038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:56:25.145515 20038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:56:25.145622 20038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:56:25.145733 20038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:56:25.155318 20038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:56:25.156508 20038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:56:25.157850 20038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:56:25.171535 20044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:25.171636 20043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:25.171613 20046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:56:25.172474 20038 server_base.cc:1034] running on GCE node
I20260929 15:56:25.173046 20038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:56:25.174014 20038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:56:25.175331 20038 hybrid_clock.cc:648] HybridClock initialized: now 1790697385175274 us; error 71 us; skew 500 ppm
I20260929 15:56:25.175668 20038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:56:25 dist-test-slave-r0ck krb5kdc[19664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697385, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:56:25.179190 20038 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:56:25.180838 20038 webserver.cc:460] Webserver started at http://127.0.97.3:44743/ using document root <none> and password file <none>
I20260929 15:56:25.181574 20038 fs_manager.cc:359] Metadata directory not provided
I20260929 15:56:25.181879 20038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:56:25.182431 20038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:56:25.186215 20038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-2/data/instance:
uuid: "28620be4d08b4068949021cecc46ce73"
format_stamp: "Formatted at 2026-09-29 15:56:25 on dist-test-slave-r0ck"
server_key: "cb1aa707fe9cf08f4e12bddb1540e538"
server_key_iv: "1fb6a90868c82705a1cd76c0889618cf"
server_key_version: "encryptionkey@0"
I20260929 15:56:25.187137 20038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "28620be4d08b4068949021cecc46ce73"
format_stamp: "Formatted at 2026-09-29 15:56:25 on dist-test-slave-r0ck"
server_key: "cb1aa707fe9cf08f4e12bddb1540e538"
server_key_iv: "1fb6a90868c82705a1cd76c0889618cf"
server_key_version: "encryptionkey@0"
I20260929 15:56:25.192757 20038 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260929 15:56:25.196249 20053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:25.197669 20038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 15:56:25.197854 20038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-2/wal
uuid: "28620be4d08b4068949021cecc46ce73"
format_stamp: "Formatted at 2026-09-29 15:56:25 on dist-test-slave-r0ck"
server_key: "cb1aa707fe9cf08f4e12bddb1540e538"
server_key_iv: "1fb6a90868c82705a1cd76c0889618cf"
server_key_version: "encryptionkey@0"
I20260929 15:56:25.198055 20038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:56:25.200124 19898 heartbeater.cc:503] Master 127.0.97.62:33557 was elected leader, sending a full tablet report...
I20260929 15:56:25.210888 20038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:56:25.215171 20038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:56:25.215593 20038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:56:25.216606 20038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:56:25.217965 20038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:56:25.218072 20038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:25.218215 20038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:56:25.218286 20038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:25.254273 20038 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45917
I20260929 15:56:25.254521 20165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45917 every 8 connection(s)
I20260929 15:56:25.255751 20038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:56:25.259083 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 20038
I20260929 15:56:25.259223 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:56:25.259895 388 external_mini_cluster.cc:1428] Setting key e1308d2dd4b6daa5643897f13f6acf12
Sep 29 15:56:25 dist-test-slave-r0ck krb5kdc[19664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697385, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:56:25.279454 20034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:25.259758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41371 (local address 127.0.97.62:33557)
0929 15:56:25.260038 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:25.260046 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:25.261839 (+ 1793us) server_negotiation.cc:407] Connection header received
0929 15:56:25.263998 (+ 2159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:25.264003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:25.264104 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:25.264226 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:25.265760 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:25.266527 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:25.268036 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:25.268466 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:25.268593 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:25.273055 (+ 4462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:25.273060 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:25.273063 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:56:25.273094 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:25.276481 (+ 3387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:25.277217 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:25.277221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:25.277223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:25.277404 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:25.277833 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:25.277836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:25.277838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:25.278095 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:25.278218 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:25.279032 (+ 814us) server_negotiation.cc:299] Negotiation successful
0929 15:56:25.279195 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260929 15:56:25.280337 20168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:25.260630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33557 (local address 127.0.97.3:41371)
0929 15:56:25.261675 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:25.261720 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:25.263730 (+ 2010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:25.264462 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:25.264470 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:25.264972 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:56:25.265595 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:25.265614 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:25.266690 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:25.266695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:25.267885 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:25.267901 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:25.268693 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:25.268700 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:25.268911 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:25.269877 (+ 966us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:56:25.269910 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:56:25.272854 (+ 2944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:56:25.276610 (+ 3756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:56:25.276617 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:56:25.276629 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:56:25.277058 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:56:25.277539 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:56:25.277543 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:56:25.277546 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:56:25.277725 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:56:25.278333 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:56:25.278339 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:56:25.278644 (+ 305us) client_negotiation.cc:769] Sending connection context
0929 15:56:25.278888 (+ 244us) client_negotiation.cc:240] Negotiation successful
0929 15:56:25.279253 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":430,"threads_started":1}
I20260929 15:56:25.282289 20166 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33557
I20260929 15:56:25.282573 20166 heartbeater.cc:464] Registering TS with master...
I20260929 15:56:25.283206 20166 heartbeater.cc:511] Master 127.0.97.62:33557 requested a full tablet report, sending...
I20260929 15:56:25.284552 19709 ts_manager.cc:194] Registered new tserver with Master: 28620be4d08b4068949021cecc46ce73 (127.0.97.3:45917)
I20260929 15:56:25.285297 19709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:41371
I20260929 15:56:25.288540 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:56:25.305838 20034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:25.292642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57204 (local address 127.0.97.62:33557)
0929 15:56:25.292887 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:25.292894 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:25.293207 (+ 313us) server_negotiation.cc:407] Connection header received
0929 15:56:25.293388 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:25.293396 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:25.293467 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:25.293560 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:25.294847 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:25.295577 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:25.296952 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:25.297426 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:25.297535 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:25.299942 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:25.299947 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:25.299950 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:56:25.299978 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:25.302769 (+ 2791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:25.303404 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:25.303408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:25.303410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:25.303527 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:25.303896 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:25.303900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:25.303902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:25.304206 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:25.304414 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:25.305074 (+ 660us) server_negotiation.cc:299] Negotiation successful
0929 15:56:25.305439 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 15:56:25.310420 19709 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:56:25.317806 19709 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:56:25.359957 20178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:25.331169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45917 (local address 127.0.0.1:51996)
0929 15:56:25.346795 (+ 15626us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:25.346872 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:25.347206 (+ 334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:25.351649 (+ 4443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:25.351653 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:25.351675 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:56:25.352473 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:25.352484 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:25.354966 (+ 2482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:25.354971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:25.356810 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:25.356822 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:25.358901 (+ 2079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:25.358909 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:25.359104 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:25.359169 (+ 65us) client_negotiation.cc:769] Sending connection context
0929 15:56:25.359316 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 15:56:25.359516 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15170,"mutex_wait_us":76,"thread_start_us":16296,"threads_started":1}
I20260929 15:56:25.360021 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:25.331584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51996 (local address 127.0.97.3:45917)
0929 15:56:25.350689 (+ 19105us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:25.350700 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:25.350745 (+ 45us) server_negotiation.cc:407] Connection header received
0929 15:56:25.350855 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:25.350864 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:25.351077 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:25.351343 (+ 266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:56:25.352686 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:25.354710 (+ 2024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:25.356990 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:25.358392 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:25.358521 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:25.358771 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:25.359511 (+ 740us) server_negotiation.cc:299] Negotiation successful
0929 15:56:25.359683 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18872,"thread_start_us":18690,"threads_started":1}
I20260929 15:56:25.360764 20180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:25.331541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34357 (local address 127.0.0.1:49628)
0929 15:56:25.349657 (+ 18116us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:25.349677 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:25.349814 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:25.351137 (+ 1323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:25.351149 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:25.351189 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:56:25.352279 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:25.352303 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:25.354882 (+ 2579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:25.354888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:25.357429 (+ 2541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:25.357449 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:25.359280 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:25.359287 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:25.359457 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:25.359505 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 15:56:25.359584 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 15:56:25.359728 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17968,"mutex_wait_us":26,"thread_start_us":17727,"threads_started":1}
I20260929 15:56:25.361588 20183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:25.333005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49868 (local address 127.0.97.2:35101)
0929 15:56:25.351080 (+ 18075us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:25.351095 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:25.352431 (+ 1336us) server_negotiation.cc:407] Connection header received
0929 15:56:25.352550 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:25.352559 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:25.352858 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:25.353075 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:56:25.353902 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:25.354995 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:25.358264 (+ 3269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:25.359439 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:25.359570 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:25.359744 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:25.361054 (+ 1310us) server_negotiation.cc:299] Negotiation successful
0929 15:56:25.361239 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17852,"thread_start_us":17731,"threads_started":1}
I20260929 15:56:25.361613 20182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:25.332652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49628 (local address 127.0.97.1:34357)
0929 15:56:25.349058 (+ 16406us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:25.349073 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:25.349949 (+ 876us) server_negotiation.cc:407] Connection header received
0929 15:56:25.350080 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:25.350089 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:25.350474 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:25.350794 (+ 320us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:56:25.352556 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:25.354711 (+ 2155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:25.357602 (+ 2891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:25.359064 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:25.359191 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:25.359399 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:25.361086 (+ 1687us) server_negotiation.cc:299] Negotiation successful
0929 15:56:25.361263 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16164,"thread_start_us":15985,"threads_started":1}
I20260929 15:56:25.362028 20181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:25.332295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35101 (local address 127.0.0.1:49868)
0929 15:56:25.352017 (+ 19722us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:25.352046 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:25.352263 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:25.353384 (+ 1121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:25.353390 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:25.353417 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:56:25.353740 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:25.353749 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:25.355174 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:25.355178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:25.358105 (+ 2927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:25.358115 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:25.359898 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:25.359907 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:25.360456 (+ 549us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:25.360486 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 15:56:25.360563 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 15:56:25.360709 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19530,"mutex_wait_us":21,"thread_start_us":19224,"threads_started":1}
I20260929 15:56:25.364053 20101 tablet_service.cc:1467] Processing CreateTablet for tablet 78253cecb22e456cbd87655bd54219bd (DEFAULT_TABLE table=test-table [id=a667e7e8767d4ff396a5af2ec2c816c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:56:25.364812 19833 tablet_service.cc:1467] Processing CreateTablet for tablet 78253cecb22e456cbd87655bd54219bd (DEFAULT_TABLE table=test-table [id=a667e7e8767d4ff396a5af2ec2c816c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:56:25.365798 20101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78253cecb22e456cbd87655bd54219bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:56:25.365399 19967 tablet_service.cc:1467] Processing CreateTablet for tablet 78253cecb22e456cbd87655bd54219bd (DEFAULT_TABLE table=test-table [id=a667e7e8767d4ff396a5af2ec2c816c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:56:25.366355 19833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78253cecb22e456cbd87655bd54219bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:56:25.366642 19967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78253cecb22e456cbd87655bd54219bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:56:25.376601 20184 tablet_bootstrap.cc:492] T 78253cecb22e456cbd87655bd54219bd P 28620be4d08b4068949021cecc46ce73: Bootstrap starting.
I20260929 15:56:25.380226 20184 tablet_bootstrap.cc:654] T 78253cecb22e456cbd87655bd54219bd P 28620be4d08b4068949021cecc46ce73: Neither blocks nor log segments found. Creating new log.
I20260929 15:56:25.380438 20185 tablet_bootstrap.cc:492] T 78253cecb22e456cbd87655bd54219bd P 0abe277da2c944ef9c91007ec94d9bf4: Bootstrap starting.
I20260929 15:56:25.382069 20184 log.cc:824] T 78253cecb22e456cbd87655bd54219bd P 28620be4d08b4068949021cecc46ce73: Log is configured to *not* fsync() on all Append() calls
I20260929 15:56:25.383833 20185 tablet_bootstrap.cc:654] T 78253cecb22e456cbd87655bd54219bd P 0abe277da2c944ef9c91007ec94d9bf4: Neither blocks nor log segments found. Creating new log.
I20260929 15:56:25.384984 20185 log.cc:824] T 78253cecb22e456cbd87655bd54219bd P 0abe277da2c944ef9c91007ec94d9bf4: Log is configured to *not* fsync() on all Append() calls
I20260929 15:56:25.385247 20184 tablet_bootstrap.cc:492] T 78253cecb22e456cbd87655bd54219bd P 28620be4d08b4068949021cecc46ce73: No bootstrap required, opened a new log
I20260929 15:56:25.385560 20184 ts_tablet_manager.cc:1397] T 78253cecb22e456cbd87655bd54219bd P 28620be4d08b4068949021cecc46ce73: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 15:56:25.383424 20186 tablet_bootstrap.cc:492] T 78253cecb22e456cbd87655bd54219bd P 2706ab21de13409bb2b2488321cddc28: Bootstrap starting.
I20260929 15:56:25.388360 20185 tablet_bootstrap.cc:492] T 78253cecb22e456cbd87655bd54219bd P 0abe277da2c944ef9c91007ec94d9bf4: No bootstrap required, opened a new log
I20260929 15:56:25.388614 20185 ts_tablet_manager.cc:1397] T 78253cecb22e456cbd87655bd54219bd P 0abe277da2c944ef9c91007ec94d9bf4: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260929 15:56:25.392151 20186 tablet_bootstrap.cc:654] T 78253cecb22e456cbd87655bd54219bd P 2706ab21de13409bb2b2488321cddc28: Neither blocks nor log segments found. Creating new log.
I20260929 15:56:25.393324 20186 log.cc:824] T 78253cecb22e456cbd87655bd54219bd P 2706ab21de13409bb2b2488321cddc28: Log is configured to *not* fsync() on all Append() calls
I20260929 15:56:25.395946 20184 raft_consensus.cc:348] T 78253cecb22e456cbd87655bd54219bd P 28620be4d08b4068949021cecc46ce73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28620be4d08b4068949021cecc46ce73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45917 } } peers { permanent_uuid: "2706ab21de13409bb2b2488321cddc28" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34357 } } peers { permanent_uuid: "0abe277da2c944ef9c91007ec94d9bf4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35101 } }
I20260929 15:56:25.396361 20186 tablet_bootstrap.cc:492] T 78253cecb22e456cbd87655bd54219bd P 2706ab21de13409bb2b2488321cddc28: No bootstrap required, opened a new log
I20260929 15:56:25.396389 20184 raft_consensus.cc:374] T 78253cecb22e456cbd87655bd54219bd P 28620be4d08b4068949021cecc46ce73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:56:25.396481 20184 raft_consensus.cc:729] T 78253cecb22e456cbd87655bd54219bd P 28620be4d08b4068949021cecc46ce73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28620be4d08b4068949021cecc46ce73, State: Initialized, Role: FOLLOWER
I20260929 15:56:25.396672 20186 ts_tablet_manager.cc:1397] T 78253cecb22e456cbd87655bd54219bd P 2706ab21de13409bb2b2488321cddc28: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.007s
I20260929 15:56:25.397101 20184 consensus_queue.cc:260] T 78253cecb22e456cbd87655bd54219bd P 28620be4d08b4068949021cecc46ce73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28620be4d08b4068949021cecc46ce73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45917 } } peers { permanent_uuid: "2706ab21de13409bb2b2488321cddc28" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34357 } } peers { permanent_uuid: "0abe277da2c944ef9c91007ec94d9bf4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35101 } }
I20260929 15:56:25.399876 20185 raft_consensus.cc:348] T 78253cecb22e456cbd87655bd54219bd P 0abe277da2c944ef9c91007ec94d9bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28620be4d08b4068949021cecc46ce73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45917 } } peers { permanent_uuid: "2706ab21de13409bb2b2488321cddc28" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34357 } } peers { permanent_uuid: "0abe277da2c944ef9c91007ec94d9bf4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35101 } }
I20260929 15:56:25.400406 20185 raft_consensus.cc:374] T 78253cecb22e456cbd87655bd54219bd P 0abe277da2c944ef9c91007ec94d9bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:56:25.400523 20185 raft_consensus.cc:729] T 78253cecb22e456cbd87655bd54219bd P 0abe277da2c944ef9c91007ec94d9bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0abe277da2c944ef9c91007ec94d9bf4, State: Initialized, Role: FOLLOWER
I20260929 15:56:25.401217 20166 heartbeater.cc:503] Master 127.0.97.62:33557 was elected leader, sending a full tablet report...
I20260929 15:56:25.401154 20185 consensus_queue.cc:260] T 78253cecb22e456cbd87655bd54219bd P 0abe277da2c944ef9c91007ec94d9bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28620be4d08b4068949021cecc46ce73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45917 } } peers { permanent_uuid: "2706ab21de13409bb2b2488321cddc28" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34357 } } peers { permanent_uuid: "0abe277da2c944ef9c91007ec94d9bf4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35101 } }
I20260929 15:56:25.402189 20184 ts_tablet_manager.cc:1428] T 78253cecb22e456cbd87655bd54219bd P 28620be4d08b4068949021cecc46ce73: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260929 15:56:25.403087 20032 heartbeater.cc:503] Master 127.0.97.62:33557 was elected leader, sending a full tablet report...
I20260929 15:56:25.403702 20185 ts_tablet_manager.cc:1428] T 78253cecb22e456cbd87655bd54219bd P 0abe277da2c944ef9c91007ec94d9bf4: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260929 15:56:25.407351 20186 raft_consensus.cc:348] T 78253cecb22e456cbd87655bd54219bd P 2706ab21de13409bb2b2488321cddc28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28620be4d08b4068949021cecc46ce73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45917 } } peers { permanent_uuid: "2706ab21de13409bb2b2488321cddc28" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34357 } } peers { permanent_uuid: "0abe277da2c944ef9c91007ec94d9bf4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35101 } }
I20260929 15:56:25.407811 20186 raft_consensus.cc:374] T 78253cecb22e456cbd87655bd54219bd P 2706ab21de13409bb2b2488321cddc28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:56:25.407905 20186 raft_consensus.cc:729] T 78253cecb22e456cbd87655bd54219bd P 2706ab21de13409bb2b2488321cddc28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2706ab21de13409bb2b2488321cddc28, State: Initialized, Role: FOLLOWER
I20260929 15:56:25.408447 20186 consensus_queue.cc:260] T 78253cecb22e456cbd87655bd54219bd P 2706ab21de13409bb2b2488321cddc28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28620be4d08b4068949021cecc46ce73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45917 } } peers { permanent_uuid: "2706ab21de13409bb2b2488321cddc28" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34357 } } peers { permanent_uuid: "0abe277da2c944ef9c91007ec94d9bf4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35101 } }
I20260929 15:56:25.409880 20186 ts_tablet_manager.cc:1428] T 78253cecb22e456cbd87655bd54219bd P 2706ab21de13409bb2b2488321cddc28: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
W20260929 15:56:25.418380 19899 tablet.cc:2369] T 78253cecb22e456cbd87655bd54219bd P 2706ab21de13409bb2b2488321cddc28: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:56:25.484158 20190 raft_consensus.cc:482] T 78253cecb22e456cbd87655bd54219bd P 28620be4d08b4068949021cecc46ce73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:56:25.484477 20190 raft_consensus.cc:504] T 78253cecb22e456cbd87655bd54219bd P 28620be4d08b4068949021cecc46ce73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28620be4d08b4068949021cecc46ce73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45917 } } peers { permanent_uuid: "2706ab21de13409bb2b2488321cddc28" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34357 } } peers { permanent_uuid: "0abe277da2c944ef9c91007ec94d9bf4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35101 } }
I20260929 15:56:25.486545 20190 leader_election.cc:288] T 78253cecb22e456cbd87655bd54219bd P 28620be4d08b4068949021cecc46ce73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2706ab21de13409bb2b2488321cddc28 (127.0.97.1:34357), 0abe277da2c944ef9c91007ec94d9bf4 (127.0.97.2:35101)
I20260929 15:56:25.494701 20182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:25.487002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55223 (local address 127.0.97.1:34357)
0929 15:56:25.487256 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:25.487260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:25.487554 (+ 294us) server_negotiation.cc:407] Connection header received
0929 15:56:25.487804 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:25.487810 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:25.487906 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:25.487999 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:56:25.488626 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:25.489775 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:25.492441 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:25.493709 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:25.493852 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:25.493952 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:25.494337 (+ 385us) server_negotiation.cc:299] Negotiation successful
0929 15:56:25.494462 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 15:56:25.494750 20168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:25.487154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34357 (local address 127.0.97.3:55223)
0929 15:56:25.487458 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:25.487476 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:25.487624 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:25.488207 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:25.488212 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:25.488267 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:56:25.488471 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:25.488481 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:25.489919 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:25.489924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:25.492270 (+ 2346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:25.492285 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:25.493899 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:25.493907 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:25.494122 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:25.494200 (+ 78us) client_negotiation.cc:769] Sending connection context
0929 15:56:25.494311 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 15:56:25.494447 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":39}
I20260929 15:56:25.495183 20183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:25.487059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58563 (local address 127.0.97.2:35101)
0929 15:56:25.487267 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:25.487273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:25.488415 (+ 1142us) server_negotiation.cc:407] Connection header received
0929 15:56:25.488617 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:25.488621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:25.488685 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:25.488786 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:56:25.489278 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:25.490284 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:25.492933 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:25.493811 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:25.493945 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:25.494042 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:25.494486 (+ 444us) server_negotiation.cc:299] Negotiation successful
0929 15:56:25.494625 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 15:56:25.495746 19853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78253cecb22e456cbd87655bd54219bd" candidate_uuid: "28620be4d08b4068949021cecc46ce73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2706ab21de13409bb2b2488321cddc28" is_pre_election: true
I20260929 15:56:25.495911 20193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:25.487473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35101 (local address 127.0.97.3:58563)
0929 15:56:25.488316 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:25.488330 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:25.488469 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:25.488801 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:25.488805 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:25.488827 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:56:25.489109 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:25.489117 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:25.490471 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:25.490477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:25.492775 (+ 2298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:25.492785 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:25.494102 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:25.494108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:25.494306 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:25.494339 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 15:56:25.495408 (+ 1069us) client_negotiation.cc:240] Negotiation successful
0929 15:56:25.495544 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"mutex_wait_us":158,"thread_start_us":439,"threads_started":1}
I20260929 15:56:25.496284 19853 raft_consensus.cc:2454] T 78253cecb22e456cbd87655bd54219bd P 2706ab21de13409bb2b2488321cddc28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28620be4d08b4068949021cecc46ce73 in term 0.
I20260929 15:56:25.496927 19987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78253cecb22e456cbd87655bd54219bd" candidate_uuid: "28620be4d08b4068949021cecc46ce73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0abe277da2c944ef9c91007ec94d9bf4" is_pre_election: true
I20260929 15:56:25.497095 20056 leader_election.cc:302] T 78253cecb22e456cbd87655bd54219bd P 28620be4d08b4068949021cecc46ce73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2706ab21de13409bb2b2488321cddc28, 28620be4d08b4068949021cecc46ce73; no voters:
I20260929 15:56:25.497457 19987 raft_consensus.cc:2454] T 78253cecb22e456cbd87655bd54219bd P 0abe277da2c944ef9c91007ec94d9bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28620be4d08b4068949021cecc46ce73 in term 0.
I20260929 15:56:25.497542 20190 raft_consensus.cc:2781] T 78253cecb22e456cbd87655bd54219bd P 28620be4d08b4068949021cecc46ce73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:56:25.497677 20190 raft_consensus.cc:482] T 78253cecb22e456cbd87655bd54219bd P 28620be4d08b4068949021cecc46ce73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:56:25.497785 20190 raft_consensus.cc:3037] T 78253cecb22e456cbd87655bd54219bd P 28620be4d08b4068949021cecc46ce73 [term 0 FOLLOWER]: Advancing to term 1
W20260929 15:56:25.498472 20033 tablet.cc:2369] T 78253cecb22e456cbd87655bd54219bd P 0abe277da2c944ef9c91007ec94d9bf4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:56:25.501022 20190 raft_consensus.cc:504] T 78253cecb22e456cbd87655bd54219bd P 28620be4d08b4068949021cecc46ce73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28620be4d08b4068949021cecc46ce73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45917 } } peers { permanent_uuid: "2706ab21de13409bb2b2488321cddc28" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34357 } } peers { permanent_uuid: "0abe277da2c944ef9c91007ec94d9bf4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35101 } }
I20260929 15:56:25.501730 20190 leader_election.cc:288] T 78253cecb22e456cbd87655bd54219bd P 28620be4d08b4068949021cecc46ce73 [CANDIDATE]: Term 1 election: Requested vote from peers 2706ab21de13409bb2b2488321cddc28 (127.0.97.1:34357), 0abe277da2c944ef9c91007ec94d9bf4 (127.0.97.2:35101)
I20260929 15:56:25.502259 19853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78253cecb22e456cbd87655bd54219bd" candidate_uuid: "28620be4d08b4068949021cecc46ce73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2706ab21de13409bb2b2488321cddc28"
I20260929 15:56:25.502501 19853 raft_consensus.cc:3037] T 78253cecb22e456cbd87655bd54219bd P 2706ab21de13409bb2b2488321cddc28 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:56:25.503659 19987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78253cecb22e456cbd87655bd54219bd" candidate_uuid: "28620be4d08b4068949021cecc46ce73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0abe277da2c944ef9c91007ec94d9bf4"
I20260929 15:56:25.503827 19987 raft_consensus.cc:3037] T 78253cecb22e456cbd87655bd54219bd P 0abe277da2c944ef9c91007ec94d9bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:56:25.505026 19853 raft_consensus.cc:2454] T 78253cecb22e456cbd87655bd54219bd P 2706ab21de13409bb2b2488321cddc28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28620be4d08b4068949021cecc46ce73 in term 1.
I20260929 15:56:25.505460 20056 leader_election.cc:302] T 78253cecb22e456cbd87655bd54219bd P 28620be4d08b4068949021cecc46ce73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2706ab21de13409bb2b2488321cddc28, 28620be4d08b4068949021cecc46ce73; no voters:
I20260929 15:56:25.505756 20190 raft_consensus.cc:2781] T 78253cecb22e456cbd87655bd54219bd P 28620be4d08b4068949021cecc46ce73 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:56:25.506211 20190 raft_consensus.cc:686] T 78253cecb22e456cbd87655bd54219bd P 28620be4d08b4068949021cecc46ce73 [term 1 LEADER]: Becoming Leader. State: Replica: 28620be4d08b4068949021cecc46ce73, State: Running, Role: LEADER
I20260929 15:56:25.506338 19987 raft_consensus.cc:2454] T 78253cecb22e456cbd87655bd54219bd P 0abe277da2c944ef9c91007ec94d9bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28620be4d08b4068949021cecc46ce73 in term 1.
I20260929 15:56:25.506832 20190 consensus_queue.cc:237] T 78253cecb22e456cbd87655bd54219bd P 28620be4d08b4068949021cecc46ce73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28620be4d08b4068949021cecc46ce73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45917 } } peers { permanent_uuid: "2706ab21de13409bb2b2488321cddc28" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34357 } } peers { permanent_uuid: "0abe277da2c944ef9c91007ec94d9bf4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35101 } }
W20260929 15:56:25.507354 20167 tablet.cc:2369] T 78253cecb22e456cbd87655bd54219bd P 28620be4d08b4068949021cecc46ce73: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:56:25.511179 19709 catalog_manager.cc:5495] T 78253cecb22e456cbd87655bd54219bd P 28620be4d08b4068949021cecc46ce73 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28620be4d08b4068949021cecc46ce73 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "28620be4d08b4068949021cecc46ce73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28620be4d08b4068949021cecc46ce73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2706ab21de13409bb2b2488321cddc28" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0abe277da2c944ef9c91007ec94d9bf4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35101 } health_report { overall_health: UNKNOWN } } }
I20260929 15:56:25.552841 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:25.547159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52004 (local address 127.0.97.3:45917)
0929 15:56:25.547390 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:25.547396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:25.547501 (+ 105us) server_negotiation.cc:407] Connection header received
0929 15:56:25.547766 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:25.547771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:25.547860 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:25.548014 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:56:25.548393 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:25.549139 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:25.550577 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:25.550987 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:25.551114 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:25.551725 (+ 611us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:56:25.552251 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:56:25.552423 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:25.552481 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 15:56:25.552560 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 15:56:25.561355 19987 raft_consensus.cc:1261] T 78253cecb22e456cbd87655bd54219bd P 0abe277da2c944ef9c91007ec94d9bf4 [term 1 FOLLOWER]: Refusing update from remote peer 28620be4d08b4068949021cecc46ce73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:56:25.561321 19853 raft_consensus.cc:1261] T 78253cecb22e456cbd87655bd54219bd P 2706ab21de13409bb2b2488321cddc28 [term 1 FOLLOWER]: Refusing update from remote peer 28620be4d08b4068949021cecc46ce73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:56:25.562528 20190 consensus_queue.cc:1035] T 78253cecb22e456cbd87655bd54219bd P 28620be4d08b4068949021cecc46ce73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0abe277da2c944ef9c91007ec94d9bf4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:56:25.562844 20194 consensus_queue.cc:1035] T 78253cecb22e456cbd87655bd54219bd P 28620be4d08b4068949021cecc46ce73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2706ab21de13409bb2b2488321cddc28" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:56:25.572825 20197 mvcc.cc:205] Tried to move back new op lower bound from 7334696491249061888 to 7334696491040137216. Current Snapshot: MvccSnapshot[applied={T|T < 7334696491249061888}]
I20260929 15:56:25.591531 19708 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:56:25.591964 19708 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:56:25.596896 19708 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 53d544cfe74841b68902563da5810801: Sending DeleteTablet for 3 replicas of tablet 78253cecb22e456cbd87655bd54219bd
I20260929 15:56:25.598001 19833 tablet_service.cc:1514] Processing DeleteTablet for tablet 78253cecb22e456cbd87655bd54219bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:56:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49628
I20260929 15:56:25.598052 20101 tablet_service.cc:1514] Processing DeleteTablet for tablet 78253cecb22e456cbd87655bd54219bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:56:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51996
I20260929 15:56:25.598214 19967 tablet_service.cc:1514] Processing DeleteTablet for tablet 78253cecb22e456cbd87655bd54219bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:56:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49868
I20260929 15:56:25.599046 20208 tablet_replica.cc:330] T 78253cecb22e456cbd87655bd54219bd P 2706ab21de13409bb2b2488321cddc28: stopping tablet replica
I20260929 15:56:25.599550 19966 tablet_service.cc:1514] Processing DeleteTablet for tablet 78253cecb22e456cbd87655bd54219bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:56:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49868
I20260929 15:56:25.599601 20208 raft_consensus.cc:2229] T 78253cecb22e456cbd87655bd54219bd P 2706ab21de13409bb2b2488321cddc28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:56:25.600623 20208 raft_consensus.cc:2258] T 78253cecb22e456cbd87655bd54219bd P 2706ab21de13409bb2b2488321cddc28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:56:25.601521 20207 tablet_replica.cc:330] T 78253cecb22e456cbd87655bd54219bd P 28620be4d08b4068949021cecc46ce73: stopping tablet replica
I20260929 15:56:25.602424 20207 raft_consensus.cc:2229] T 78253cecb22e456cbd87655bd54219bd P 28620be4d08b4068949021cecc46ce73 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:56:25.603180 20207 raft_consensus.cc:2258] T 78253cecb22e456cbd87655bd54219bd P 28620be4d08b4068949021cecc46ce73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:56:25.603888 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 19770
I20260929 15:56:25.605460 20207 ts_tablet_manager.cc:1905] T 78253cecb22e456cbd87655bd54219bd P 28620be4d08b4068949021cecc46ce73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:56:25.613516 20207 ts_tablet_manager.cc:1918] T 78253cecb22e456cbd87655bd54219bd P 28620be4d08b4068949021cecc46ce73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:56:25.614463 20207 log.cc:1196] T 78253cecb22e456cbd87655bd54219bd P 28620be4d08b4068949021cecc46ce73: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-2/wal/wals/78253cecb22e456cbd87655bd54219bd
I20260929 15:56:25.615334 20209 tablet_replica.cc:330] T 78253cecb22e456cbd87655bd54219bd P 0abe277da2c944ef9c91007ec94d9bf4: stopping tablet replica
I20260929 15:56:25.615813 20209 raft_consensus.cc:2229] T 78253cecb22e456cbd87655bd54219bd P 0abe277da2c944ef9c91007ec94d9bf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:56:25.616132 20207 ts_tablet_manager.cc:1939] T 78253cecb22e456cbd87655bd54219bd P 28620be4d08b4068949021cecc46ce73: Deleting consensus metadata
I20260929 15:56:25.616344 20209 raft_consensus.cc:2258] T 78253cecb22e456cbd87655bd54219bd P 0abe277da2c944ef9c91007ec94d9bf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:56:25.616537 20208 ts_tablet_manager.cc:1905] T 78253cecb22e456cbd87655bd54219bd P 2706ab21de13409bb2b2488321cddc28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:56:25.618317 19694 catalog_manager.cc:4841] TS 28620be4d08b4068949021cecc46ce73 (127.0.97.3:45917): tablet 78253cecb22e456cbd87655bd54219bd (table test-table [id=a667e7e8767d4ff396a5af2ec2c816c8]) successfully deleted
I20260929 15:56:25.618505 20209 ts_tablet_manager.cc:1905] T 78253cecb22e456cbd87655bd54219bd P 0abe277da2c944ef9c91007ec94d9bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:56:25.624256 20208 ts_tablet_manager.cc:1918] T 78253cecb22e456cbd87655bd54219bd P 2706ab21de13409bb2b2488321cddc28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:56:25.624425 20208 log.cc:1196] T 78253cecb22e456cbd87655bd54219bd P 2706ab21de13409bb2b2488321cddc28: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-0/wal/wals/78253cecb22e456cbd87655bd54219bd
I20260929 15:56:25.624949 20208 ts_tablet_manager.cc:1939] T 78253cecb22e456cbd87655bd54219bd P 2706ab21de13409bb2b2488321cddc28: Deleting consensus metadata
Sep 29 15:56:25 dist-test-slave-r0ck krb5kdc[19664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 15:56:25.625470 20209 ts_tablet_manager.cc:1918] T 78253cecb22e456cbd87655bd54219bd P 0abe277da2c944ef9c91007ec94d9bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:56:25.625629 20209 log.cc:1196] T 78253cecb22e456cbd87655bd54219bd P 0abe277da2c944ef9c91007ec94d9bf4: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697075657932-388-0/minicluster-data/ts-1/wal/wals/78253cecb22e456cbd87655bd54219bd
I20260929 15:56:25.626154 20209 ts_tablet_manager.cc:1939] T 78253cecb22e456cbd87655bd54219bd P 0abe277da2c944ef9c91007ec94d9bf4: Deleting consensus metadata
I20260929 15:56:25.626778 19695 catalog_manager.cc:4841] TS 2706ab21de13409bb2b2488321cddc28 (127.0.97.1:34357): tablet 78253cecb22e456cbd87655bd54219bd (table test-table [id=a667e7e8767d4ff396a5af2ec2c816c8]) successfully deleted
I20260929 15:56:25.627837 19693 catalog_manager.cc:4841] TS 0abe277da2c944ef9c91007ec94d9bf4 (127.0.97.2:35101): tablet 78253cecb22e456cbd87655bd54219bd (table test-table [id=a667e7e8767d4ff396a5af2ec2c816c8]) successfully deleted
W20260929 15:56:25.628278 19693 catalog_manager.cc:4808] TS 0abe277da2c944ef9c91007ec94d9bf4 (127.0.97.2:35101): delete failed for tablet 78253cecb22e456cbd87655bd54219bd because the tablet was not found. No further retry: Not found: Tablet not found: 78253cecb22e456cbd87655bd54219bd
I20260929 15:56:25.633126 20182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:25.604937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49640 (local address 127.0.97.1:34357)
0929 15:56:25.605159 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:25.605168 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:25.614333 (+ 9165us) server_negotiation.cc:407] Connection header received
0929 15:56:25.614435 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:25.614444 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:25.614569 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:25.614720 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:25.616383 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:25.617531 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:25.620024 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:25.620654 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:25.620768 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:25.626408 (+ 5640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:25.626422 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:25.626438 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:56:25.626485 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:25.629764 (+ 3279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:25.630663 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:25.630674 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:25.630676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:25.630947 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:25.631534 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:25.631539 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:25.631541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:25.631965 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:25.632268 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:25.632583 (+ 315us) server_negotiation.cc:299] Negotiation successful
0929 15:56:25.632802 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 15:56:25.633854 19893 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:56:25.722891 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 19770
I20260929 15:56:25.747237 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 19904
Sep 29 15:56:25 dist-test-slave-r0ck krb5kdc[19664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 15:56:25.767625 20183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:25.748051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49870 (local address 127.0.97.2:35101)
0929 15:56:25.748265 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:25.748272 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:25.748327 (+ 55us) server_negotiation.cc:407] Connection header received
0929 15:56:25.748515 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:25.748521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:25.748612 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:25.748723 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:25.750595 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:25.751740 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:25.753562 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:25.754231 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:25.754357 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:25.759784 (+ 5427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:25.759803 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:25.759824 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:56:25.759869 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:25.763801 (+ 3932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:25.764628 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:25.764638 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:25.764642 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:25.764783 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:25.765173 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:25.765178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:25.765182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:25.765668 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:25.765970 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:25.767044 (+ 1074us) server_negotiation.cc:299] Negotiation successful
0929 15:56:25.767244 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 15:56:25.768344 20027 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:56:25.862198 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 19904
I20260929 15:56:25.889051 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 20038
Sep 29 15:56:25 dist-test-slave-r0ck krb5kdc[19664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 15:56:25.908460 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:25.889872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52008 (local address 127.0.97.3:45917)
0929 15:56:25.890067 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:25.890072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:25.890157 (+ 85us) server_negotiation.cc:407] Connection header received
0929 15:56:25.890352 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:25.890356 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:25.890417 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:25.890505 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:25.892379 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:25.893723 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:25.895741 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:25.896260 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:25.896383 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:25.902090 (+ 5707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:25.902104 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:25.902120 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:56:25.902164 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:25.905325 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:25.906153 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:25.906166 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:25.906169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:25.906342 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:25.906756 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:25.906762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:25.906765 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:25.907129 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:25.907512 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:25.907824 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 15:56:25.908054 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 15:56:25.909118 20161 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:56:26.044452 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 20038
I20260929 15:56:26.067130 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 19677
I20260929 15:56:26.068379 19739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:56:26.192024 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 19677
2026-09-29T15:56:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4572 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T15:56:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:56:26Z Disabled control of system clock
I20260929 15:56:26.271238 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40595
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:37237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 15:56:26.600685 20218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:56:26.600948 20218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:56:26.601078 20218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:56:26.608507 20218 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:56:26.608629 20218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:56:26.608668 20218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:56:26.608776 20218 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:56:26.608857 20218 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:56:26.618444 20218 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40595
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:56:26.619987 20218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:56:26.621798 20218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:56:26.633713 20223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:26.633787 20224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:26.633782 20226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:56:26.634508 20218 server_base.cc:1034] running on GCE node
I20260929 15:56:26.635236 20218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:56:26.636584 20218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:56:26.637804 20218 hybrid_clock.cc:648] HybridClock initialized: now 1790697386637771 us; error 32 us; skew 500 ppm
I20260929 15:56:26.638160 20218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:56:26.640707 20218 webserver.cc:460] Webserver started at http://127.0.97.62:39633/ using document root <none> and password file <none>
I20260929 15:56:26.641417 20218 fs_manager.cc:359] Metadata directory not provided
I20260929 15:56:26.641513 20218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:56:26.641903 20218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:56:26.644675 20218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "619707b8e55042a29116bc9486a86f99"
format_stamp: "Formatted at 2026-09-29 15:56:26 on dist-test-slave-r0ck"
server_key: "c5671d3b86bd50e9aea7018e45494d27"
server_key_iv: "c1d1b877e2941ae93c725252266b47e0"
server_key_version: "encryptionkey@0"
I20260929 15:56:26.645520 20218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "619707b8e55042a29116bc9486a86f99"
format_stamp: "Formatted at 2026-09-29 15:56:26 on dist-test-slave-r0ck"
server_key: "c5671d3b86bd50e9aea7018e45494d27"
server_key_iv: "c1d1b877e2941ae93c725252266b47e0"
server_key_version: "encryptionkey@0"
I20260929 15:56:26.650588 20218 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 15:56:26.654438 20232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:26.656486 20218 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 15:56:26.656719 20218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "619707b8e55042a29116bc9486a86f99"
format_stamp: "Formatted at 2026-09-29 15:56:26 on dist-test-slave-r0ck"
server_key: "c5671d3b86bd50e9aea7018e45494d27"
server_key_iv: "c1d1b877e2941ae93c725252266b47e0"
server_key_version: "encryptionkey@0"
I20260929 15:56:26.656965 20218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:56:26.678722 20218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:56:26.679688 20218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:56:26.679984 20218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:56:26.698498 20218 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40595
I20260929 15:56:26.698518 20283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40595 every 8 connection(s)
I20260929 15:56:26.699832 20218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:56:26.702919 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 20218
I20260929 15:56:26.703112 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:56:26.703586 388 external_mini_cluster.cc:1428] Setting key ef4d3711ac977ac3848d2ba46f63670d
I20260929 15:56:26.703588 20284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:56:26.718271 20287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:26.705382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49032 (local address 127.0.97.62:40595)
0929 15:56:26.706435 (+ 1053us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:26.706452 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:26.706501 (+ 49us) server_negotiation.cc:407] Connection header received
0929 15:56:26.708511 (+ 2010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:26.708536 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:26.709079 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:26.709513 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:26.710031 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:26.712013 (+ 1982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:26.713337 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:26.714359 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:26.714524 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:26.715604 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:26.715614 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:26.715630 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:26.716343 (+ 713us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:26.716431 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:26.716456 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:26.716587 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:26.716801 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 15:56:26.717082 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":342,"threads_started":1}
I20260929 15:56:26.718739 20284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 619707b8e55042a29116bc9486a86f99: Bootstrap starting.
I20260929 15:56:26.721675 20284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 619707b8e55042a29116bc9486a86f99: Neither blocks nor log segments found. Creating new log.
I20260929 15:56:26.722599 20284 log.cc:824] T 00000000000000000000000000000000 P 619707b8e55042a29116bc9486a86f99: Log is configured to *not* fsync() on all Append() calls
I20260929 15:56:26.725412 20284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 619707b8e55042a29116bc9486a86f99: No bootstrap required, opened a new log
I20260929 15:56:26.732501 20284 raft_consensus.cc:348] T 00000000000000000000000000000000 P 619707b8e55042a29116bc9486a86f99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619707b8e55042a29116bc9486a86f99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40595 } }
I20260929 15:56:26.732851 20284 raft_consensus.cc:374] T 00000000000000000000000000000000 P 619707b8e55042a29116bc9486a86f99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:56:26.732955 20284 raft_consensus.cc:729] T 00000000000000000000000000000000 P 619707b8e55042a29116bc9486a86f99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 619707b8e55042a29116bc9486a86f99, State: Initialized, Role: FOLLOWER
I20260929 15:56:26.733530 20284 consensus_queue.cc:260] T 00000000000000000000000000000000 P 619707b8e55042a29116bc9486a86f99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619707b8e55042a29116bc9486a86f99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40595 } }
I20260929 15:56:26.733770 20284 raft_consensus.cc:388] T 00000000000000000000000000000000 P 619707b8e55042a29116bc9486a86f99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:56:26.733922 20284 raft_consensus.cc:482] T 00000000000000000000000000000000 P 619707b8e55042a29116bc9486a86f99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:56:26.734086 20284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 619707b8e55042a29116bc9486a86f99 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:56:26.736151 20284 raft_consensus.cc:504] T 00000000000000000000000000000000 P 619707b8e55042a29116bc9486a86f99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619707b8e55042a29116bc9486a86f99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40595 } }
I20260929 15:56:26.736662 20284 leader_election.cc:302] T 00000000000000000000000000000000 P 619707b8e55042a29116bc9486a86f99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 619707b8e55042a29116bc9486a86f99; no voters:
I20260929 15:56:26.737417 20284 leader_election.cc:288] T 00000000000000000000000000000000 P 619707b8e55042a29116bc9486a86f99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:56:26.737531 20289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 619707b8e55042a29116bc9486a86f99 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:56:26.738288 20289 raft_consensus.cc:686] T 00000000000000000000000000000000 P 619707b8e55042a29116bc9486a86f99 [term 1 LEADER]: Becoming Leader. State: Replica: 619707b8e55042a29116bc9486a86f99, State: Running, Role: LEADER
I20260929 15:56:26.738759 20289 consensus_queue.cc:237] T 00000000000000000000000000000000 P 619707b8e55042a29116bc9486a86f99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619707b8e55042a29116bc9486a86f99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40595 } }
I20260929 15:56:26.739400 20284 sys_catalog.cc:564] T 00000000000000000000000000000000 P 619707b8e55042a29116bc9486a86f99 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:56:26.741786 20291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 619707b8e55042a29116bc9486a86f99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 619707b8e55042a29116bc9486a86f99. Latest consensus state: current_term: 1 leader_uuid: "619707b8e55042a29116bc9486a86f99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619707b8e55042a29116bc9486a86f99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40595 } } }
I20260929 15:56:26.742148 20291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 619707b8e55042a29116bc9486a86f99 [sys.catalog]: This master's current role is: LEADER
I20260929 15:56:26.741883 20290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 619707b8e55042a29116bc9486a86f99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "619707b8e55042a29116bc9486a86f99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619707b8e55042a29116bc9486a86f99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40595 } } }
I20260929 15:56:26.743618 20290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 619707b8e55042a29116bc9486a86f99 [sys.catalog]: This master's current role is: LEADER
I20260929 15:56:26.744174 20295 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:56:26.749485 20295 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:56:26.758320 20295 catalog_manager.cc:1347] Generated new cluster ID: 3c8f13498a234f1093de0ff35be6bd67
I20260929 15:56:26.758440 20295 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:56:26.771872 20295 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:56:26.773128 20295 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:56:26.785630 20295 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 619707b8e55042a29116bc9486a86f99: Generated new TSK 0
I20260929 15:56:26.786567 20295 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:56:26.804340 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40595
--builtin_ntp_servers=127.0.97.20:37237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 15:56:27.140019 20308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:56:27.140333 20308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:56:27.140532 20308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:56:27.148186 20308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:56:27.148334 20308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:56:27.148518 20308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:56:27.158043 20308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:56:27.159370 20308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:56:27.161237 20308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:56:27.174527 20313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:27.174614 20316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:27.174664 20314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:56:27.175963 20308 server_base.cc:1034] running on GCE node
I20260929 15:56:27.176649 20308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:56:27.177671 20308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:56:27.178939 20308 hybrid_clock.cc:648] HybridClock initialized: now 1790697387178876 us; error 44 us; skew 500 ppm
I20260929 15:56:27.179313 20308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:56:27.181795 20308 webserver.cc:460] Webserver started at http://127.0.97.1:40019/ using document root <none> and password file <none>
I20260929 15:56:27.182499 20308 fs_manager.cc:359] Metadata directory not provided
I20260929 15:56:27.182633 20308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:56:27.182951 20308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:56:27.185989 20308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-0/data/instance:
uuid: "f9d2efa77e03489cb9b391806e126d71"
format_stamp: "Formatted at 2026-09-29 15:56:27 on dist-test-slave-r0ck"
server_key: "2cd0e1e966b59f66d10a5c05c82c6f4c"
server_key_iv: "16ad5a8f437589d7b76d37e45115d116"
server_key_version: "encryptionkey@0"
I20260929 15:56:27.186779 20308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f9d2efa77e03489cb9b391806e126d71"
format_stamp: "Formatted at 2026-09-29 15:56:27 on dist-test-slave-r0ck"
server_key: "2cd0e1e966b59f66d10a5c05c82c6f4c"
server_key_iv: "16ad5a8f437589d7b76d37e45115d116"
server_key_version: "encryptionkey@0"
I20260929 15:56:27.191838 20308 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 15:56:27.195309 20322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:27.196643 20308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 15:56:27.196828 20308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-0/wal
uuid: "f9d2efa77e03489cb9b391806e126d71"
format_stamp: "Formatted at 2026-09-29 15:56:27 on dist-test-slave-r0ck"
server_key: "2cd0e1e966b59f66d10a5c05c82c6f4c"
server_key_iv: "16ad5a8f437589d7b76d37e45115d116"
server_key_version: "encryptionkey@0"
I20260929 15:56:27.197053 20308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:56:27.228607 20308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:56:27.229511 20308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:56:27.229771 20308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:56:27.230710 20308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:56:27.232156 20308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:56:27.232290 20308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:27.232414 20308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:56:27.232499 20308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:27.266742 20308 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43169
I20260929 15:56:27.266832 20434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43169 every 8 connection(s)
I20260929 15:56:27.268024 20308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:56:27.271965 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 20308
I20260929 15:56:27.272156 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:56:27.272687 388 external_mini_cluster.cc:1428] Setting key 06facbc34c9fb54cfb20762fe2064566
I20260929 15:56:27.276166 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40595
--builtin_ntp_servers=127.0.97.20:37237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 15:56:27.282361 20437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:27.270707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47875 (local address 127.0.97.62:40595)
0929 15:56:27.271629 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:27.271639 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:27.272727 (+ 1088us) server_negotiation.cc:407] Connection header received
0929 15:56:27.275199 (+ 2472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:27.275204 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:27.275299 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:27.275420 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:27.276534 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:27.277271 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:27.278902 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:27.279313 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:27.279448 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:27.281145 (+ 1697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:27.281152 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:27.281161 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:27.281191 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:27.281251 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:27.281268 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:27.281522 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:27.281818 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 15:56:27.281959 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":461,"threads_started":1}
I20260929 15:56:27.283679 20438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:27.271107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40595 (local address 127.0.97.1:47875)
0929 15:56:27.272538 (+ 1431us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:27.272578 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:27.274879 (+ 2301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:27.275677 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:27.275686 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:27.275778 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:56:27.276362 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:27.276379 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:27.277498 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:27.277503 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:27.278740 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:27.278753 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:27.279568 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:27.279578 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:27.279795 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:27.280557 (+ 762us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:56:27.280560 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:56:27.280600 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:56:27.280690 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:56:27.281529 (+ 839us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:56:27.281550 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:56:27.281561 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 15:56:27.281809 (+ 248us) client_negotiation.cc:240] Negotiation successful
0929 15:56:27.282415 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"thread_start_us":485,"threads_started":1}
I20260929 15:56:27.285521 20435 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40595
I20260929 15:56:27.285873 20435 heartbeater.cc:464] Registering TS with master...
I20260929 15:56:27.286800 20435 heartbeater.cc:511] Master 127.0.97.62:40595 requested a full tablet report, sending...
I20260929 15:56:27.289203 20249 ts_manager.cc:194] Registered new tserver with Master: f9d2efa77e03489cb9b391806e126d71 (127.0.97.1:43169)
I20260929 15:56:27.291288 20249 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:47875
W20260929 15:56:27.602032 20439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:56:27.602288 20439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:56:27.602371 20439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:56:27.609212 20439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:56:27.609294 20439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:56:27.609452 20439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:56:27.618072 20439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:56:27.619215 20439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:56:27.620834 20439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:56:27.633705 20444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:27.633728 20445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:27.633780 20447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:56:27.634517 20439 server_base.cc:1034] running on GCE node
I20260929 15:56:27.635142 20439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:56:27.636029 20439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:56:27.637265 20439 hybrid_clock.cc:648] HybridClock initialized: now 1790697387637218 us; error 55 us; skew 500 ppm
I20260929 15:56:27.637588 20439 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:56:27.639745 20439 webserver.cc:460] Webserver started at http://127.0.97.2:45949/ using document root <none> and password file <none>
I20260929 15:56:27.640457 20439 fs_manager.cc:359] Metadata directory not provided
I20260929 15:56:27.640586 20439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:56:27.640910 20439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:56:27.643787 20439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-1/data/instance:
uuid: "d1c2b3a38f26484cb4cf982abf0b1e94"
format_stamp: "Formatted at 2026-09-29 15:56:27 on dist-test-slave-r0ck"
server_key: "9f87610a7a9ade8d16380b385138315d"
server_key_iv: "d762c68a3d9fbda5a6c23c01c1ce2b09"
server_key_version: "encryptionkey@0"
I20260929 15:56:27.644541 20439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d1c2b3a38f26484cb4cf982abf0b1e94"
format_stamp: "Formatted at 2026-09-29 15:56:27 on dist-test-slave-r0ck"
server_key: "9f87610a7a9ade8d16380b385138315d"
server_key_iv: "d762c68a3d9fbda5a6c23c01c1ce2b09"
server_key_version: "encryptionkey@0"
I20260929 15:56:27.649618 20439 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 15:56:27.652945 20453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:27.654305 20439 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:56:27.654479 20439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-1/wal
uuid: "d1c2b3a38f26484cb4cf982abf0b1e94"
format_stamp: "Formatted at 2026-09-29 15:56:27 on dist-test-slave-r0ck"
server_key: "9f87610a7a9ade8d16380b385138315d"
server_key_iv: "d762c68a3d9fbda5a6c23c01c1ce2b09"
server_key_version: "encryptionkey@0"
I20260929 15:56:27.654703 20439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:56:27.683677 20439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:56:27.684667 20439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:56:27.684939 20439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:56:27.685906 20439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:56:27.687350 20439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:56:27.687471 20439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:27.687610 20439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:56:27.687703 20439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:27.723435 20439 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36111
I20260929 15:56:27.723520 20565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36111 every 8 connection(s)
I20260929 15:56:27.724687 20439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:56:27.725234 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 20439
I20260929 15:56:27.725454 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:56:27.726148 388 external_mini_cluster.cc:1428] Setting key b5ad4b2050b0f4a73c1221127b121b77
I20260929 15:56:27.729708 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40595
--builtin_ntp_servers=127.0.97.20:37237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 15:56:27.740821 20437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:27.727582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38777 (local address 127.0.97.62:40595)
0929 15:56:27.727754 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:27.727764 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:27.729507 (+ 1743us) server_negotiation.cc:407] Connection header received
0929 15:56:27.732393 (+ 2886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:27.732397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:27.732468 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:27.732559 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:27.733939 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:27.734716 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:27.737189 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:27.737818 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:27.737940 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:27.739763 (+ 1823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:27.739772 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:27.739780 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:27.739814 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:27.739857 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:27.739875 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:27.740085 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:27.740270 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 15:56:27.740407 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 15:56:27.741814 20568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:27.728228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40595 (local address 127.0.97.2:38777)
0929 15:56:27.729332 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:27.729415 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:27.732086 (+ 2671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:27.732827 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:27.732839 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:27.732954 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:56:27.733744 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:27.733780 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:27.734883 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:27.734892 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:27.737041 (+ 2149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:27.737059 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:27.738039 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:27.738048 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:27.738373 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:27.739213 (+ 840us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:56:27.739215 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:56:27.739253 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:56:27.739396 (+ 143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:56:27.740065 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:56:27.740073 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:56:27.740079 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 15:56:27.740225 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 15:56:27.740687 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":351,"threads_started":1}
I20260929 15:56:27.743252 20566 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40595
I20260929 15:56:27.743558 20566 heartbeater.cc:464] Registering TS with master...
I20260929 15:56:27.744242 20566 heartbeater.cc:511] Master 127.0.97.62:40595 requested a full tablet report, sending...
I20260929 15:56:27.745707 20249 ts_manager.cc:194] Registered new tserver with Master: d1c2b3a38f26484cb4cf982abf0b1e94 (127.0.97.2:36111)
I20260929 15:56:27.746544 20249 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:38777
W20260929 15:56:28.060559 20569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:56:28.060827 20569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:56:28.060990 20569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:56:28.068152 20569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:56:28.068244 20569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:56:28.068445 20569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:56:28.077527 20569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:56:28.078774 20569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:56:28.080502 20569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:56:28.094033 20574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:28.094172 20577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:28.094379 20575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:56:28.095176 20569 server_base.cc:1034] running on GCE node
I20260929 15:56:28.095757 20569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:56:28.096696 20569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:56:28.098026 20569 hybrid_clock.cc:648] HybridClock initialized: now 1790697388097956 us; error 60 us; skew 500 ppm
I20260929 15:56:28.098539 20569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:56:28.101475 20569 webserver.cc:460] Webserver started at http://127.0.97.3:45963/ using document root <none> and password file <none>
I20260929 15:56:28.102196 20569 fs_manager.cc:359] Metadata directory not provided
I20260929 15:56:28.102335 20569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:56:28.102659 20569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:56:28.105500 20569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-2/data/instance:
uuid: "feaa4e02253c4e2c883fc46f12806187"
format_stamp: "Formatted at 2026-09-29 15:56:28 on dist-test-slave-r0ck"
server_key: "8c68b6cbc705c103be2e79873a6ae5ea"
server_key_iv: "c382768efc712c95fc3d1b88fb74fd56"
server_key_version: "encryptionkey@0"
I20260929 15:56:28.106283 20569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-2/wal/instance:
uuid: "feaa4e02253c4e2c883fc46f12806187"
format_stamp: "Formatted at 2026-09-29 15:56:28 on dist-test-slave-r0ck"
server_key: "8c68b6cbc705c103be2e79873a6ae5ea"
server_key_iv: "c382768efc712c95fc3d1b88fb74fd56"
server_key_version: "encryptionkey@0"
I20260929 15:56:28.111145 20569 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 15:56:28.114578 20583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:28.116072 20569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:56:28.116266 20569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-2/wal
uuid: "feaa4e02253c4e2c883fc46f12806187"
format_stamp: "Formatted at 2026-09-29 15:56:28 on dist-test-slave-r0ck"
server_key: "8c68b6cbc705c103be2e79873a6ae5ea"
server_key_iv: "c382768efc712c95fc3d1b88fb74fd56"
server_key_version: "encryptionkey@0"
I20260929 15:56:28.116511 20569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:56:28.131793 20569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:56:28.132702 20569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:56:28.132998 20569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:56:28.133946 20569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:56:28.135458 20569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:56:28.135576 20569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:28.135710 20569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:56:28.135807 20569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:28.168843 20569 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37287
I20260929 15:56:28.168901 20695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37287 every 8 connection(s)
I20260929 15:56:28.170269 20569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:56:28.178035 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 20569
I20260929 15:56:28.178205 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:56:28.178722 388 external_mini_cluster.cc:1428] Setting key a6429ce1ed2feb29940453ad1040cfc0
I20260929 15:56:28.183246 20437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:28.172872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48961 (local address 127.0.97.62:40595)
0929 15:56:28.173108 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:28.173115 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:28.174523 (+ 1408us) server_negotiation.cc:407] Connection header received
0929 15:56:28.176215 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:28.176219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:28.176285 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:28.176384 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:28.177481 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:28.178645 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:28.180505 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:28.180922 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:28.181056 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:28.182457 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:28.182462 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:28.182465 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:28.182482 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:28.182511 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:28.182522 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:28.182619 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:28.182843 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 15:56:28.182932 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 15:56:28.184130 20698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:28.173466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40595 (local address 127.0.97.3:48961)
0929 15:56:28.174382 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:28.174417 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:28.175965 (+ 1548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:28.176617 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:28.176625 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:28.176705 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:56:28.177262 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:28.177278 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:28.178816 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:28.178824 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:28.180367 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:28.180380 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:28.181151 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:28.181158 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:28.181459 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:28.182147 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:56:28.182150 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:56:28.182185 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:56:28.182262 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:56:28.182618 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:56:28.182629 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:56:28.182639 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 15:56:28.182812 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 15:56:28.183171 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":291,"threads_started":1}
I20260929 15:56:28.185320 20696 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40595
I20260929 15:56:28.185684 20696 heartbeater.cc:464] Registering TS with master...
I20260929 15:56:28.186432 20696 heartbeater.cc:511] Master 127.0.97.62:40595 requested a full tablet report, sending...
I20260929 15:56:28.188107 20249 ts_manager.cc:194] Registered new tserver with Master: feaa4e02253c4e2c883fc46f12806187 (127.0.97.3:37287)
I20260929 15:56:28.188923 20249 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:48961
I20260929 15:56:28.193735 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:56:28.203591 20437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:28.197756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53334 (local address 127.0.97.62:40595)
0929 15:56:28.197926 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:28.197930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:28.198334 (+ 404us) server_negotiation.cc:407] Connection header received
0929 15:56:28.198466 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:28.198469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:28.198542 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:28.198627 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:28.199054 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:28.199802 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:28.201438 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:28.201919 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:28.202020 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:28.202985 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:28.202990 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:28.202993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:28.203009 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:28.203038 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:28.203049 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:28.203121 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:28.203303 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 15:56:28.203376 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 15:56:28.207306 20249 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53334:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:56:28.214214 20249 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:56:28.238849 20710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:28.226624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43169 (local address 127.0.0.1:58776)
0929 15:56:28.229189 (+ 2565us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:28.229212 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:28.229385 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:28.230242 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:28.230253 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:28.230297 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:56:28.231261 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:28.231278 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:28.233441 (+ 2163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:28.233453 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:28.236104 (+ 2651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:28.236118 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:28.237869 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:28.237875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:28.238051 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:28.238102 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 15:56:28.238257 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 15:56:28.238451 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2399,"mutex_wait_us":34,"thread_start_us":2468,"threads_started":1}
I20260929 15:56:28.239949 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:28.226783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58776 (local address 127.0.97.1:43169)
0929 15:56:28.228725 (+ 1942us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:28.228743 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:28.229476 (+ 733us) server_negotiation.cc:407] Connection header received
0929 15:56:28.229612 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:28.229624 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:28.229940 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:28.231510 (+ 1570us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:56:28.231791 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:28.233203 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:28.236311 (+ 3108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:28.237691 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:28.237842 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:28.238091 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:28.239353 (+ 1262us) server_negotiation.cc:299] Negotiation successful
0929 15:56:28.239551 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1646,"thread_start_us":1555,"threads_started":1}
I20260929 15:56:28.240211 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:28.228388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37287 (local address 127.0.0.1:58328)
0929 15:56:28.232064 (+ 3676us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:28.232086 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:28.232214 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:28.232691 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:28.232695 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:28.232720 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:56:28.233108 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:28.233122 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:28.235146 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:28.235151 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:28.237297 (+ 2146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:28.237312 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:28.239352 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:28.239359 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:28.239595 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:28.239637 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 15:56:28.239740 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 15:56:28.239864 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3468,"mutex_wait_us":111,"thread_start_us":3318,"threads_started":1}
I20260929 15:56:28.240331 20708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:28.226223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36111 (local address 127.0.0.1:43384)
0929 15:56:28.227083 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:28.227131 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:28.227344 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:28.232995 (+ 5651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:28.232999 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:28.233030 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:56:28.233523 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:28.233536 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:28.235299 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:28.235305 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:28.237090 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:28.237101 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:28.239352 (+ 2251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:28.239359 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:28.239612 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:28.239657 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 15:56:28.239792 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 15:56:28.239943 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"mutex_wait_us":116,"thread_start_us":374,"threads_started":1}
I20260929 15:56:28.240815 20713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:28.228760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58328 (local address 127.0.97.3:37287)
0929 15:56:28.230166 (+ 1406us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:28.230177 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:28.232241 (+ 2064us) server_negotiation.cc:407] Connection header received
0929 15:56:28.232374 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:28.232379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:28.232523 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:28.232700 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:56:28.233330 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:28.234982 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:28.237490 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:28.238855 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:28.238985 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:28.239222 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:28.239809 (+ 587us) server_negotiation.cc:299] Negotiation successful
0929 15:56:28.239957 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1009,"thread_start_us":872,"threads_started":1}
I20260929 15:56:28.241212 20709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:28.226452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43384 (local address 127.0.97.2:36111)
0929 15:56:28.232289 (+ 5837us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:28.232296 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:28.232324 (+ 28us) server_negotiation.cc:407] Connection header received
0929 15:56:28.232408 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:28.232415 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:28.232601 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:28.232790 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:56:28.233709 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:28.235155 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:28.237271 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:28.238688 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:28.238836 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:28.239568 (+ 732us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:28.240554 (+ 986us) server_negotiation.cc:299] Negotiation successful
0929 15:56:28.240708 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5677,"thread_start_us":5547,"threads_started":1}
I20260929 15:56:28.244279 20370 tablet_service.cc:1467] Processing CreateTablet for tablet a2d9e3c265574aafaadb2fcd22b5adee (DEFAULT_TABLE table=test-table [id=2ca5f044775149799114ec4451c8ce05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:56:28.244738 20501 tablet_service.cc:1467] Processing CreateTablet for tablet a2d9e3c265574aafaadb2fcd22b5adee (DEFAULT_TABLE table=test-table [id=2ca5f044775149799114ec4451c8ce05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:56:28.244851 20631 tablet_service.cc:1467] Processing CreateTablet for tablet a2d9e3c265574aafaadb2fcd22b5adee (DEFAULT_TABLE table=test-table [id=2ca5f044775149799114ec4451c8ce05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:56:28.245796 20370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2d9e3c265574aafaadb2fcd22b5adee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:56:28.245908 20501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2d9e3c265574aafaadb2fcd22b5adee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:56:28.246148 20631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2d9e3c265574aafaadb2fcd22b5adee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:56:28.258351 20714 tablet_bootstrap.cc:492] T a2d9e3c265574aafaadb2fcd22b5adee P feaa4e02253c4e2c883fc46f12806187: Bootstrap starting.
I20260929 15:56:28.260318 20715 tablet_bootstrap.cc:492] T a2d9e3c265574aafaadb2fcd22b5adee P f9d2efa77e03489cb9b391806e126d71: Bootstrap starting.
I20260929 15:56:28.262442 20714 tablet_bootstrap.cc:654] T a2d9e3c265574aafaadb2fcd22b5adee P feaa4e02253c4e2c883fc46f12806187: Neither blocks nor log segments found. Creating new log.
I20260929 15:56:28.263854 20714 log.cc:824] T a2d9e3c265574aafaadb2fcd22b5adee P feaa4e02253c4e2c883fc46f12806187: Log is configured to *not* fsync() on all Append() calls
I20260929 15:56:28.263957 20715 tablet_bootstrap.cc:654] T a2d9e3c265574aafaadb2fcd22b5adee P f9d2efa77e03489cb9b391806e126d71: Neither blocks nor log segments found. Creating new log.
I20260929 15:56:28.265183 20715 log.cc:824] T a2d9e3c265574aafaadb2fcd22b5adee P f9d2efa77e03489cb9b391806e126d71: Log is configured to *not* fsync() on all Append() calls
I20260929 15:56:28.267151 20716 tablet_bootstrap.cc:492] T a2d9e3c265574aafaadb2fcd22b5adee P d1c2b3a38f26484cb4cf982abf0b1e94: Bootstrap starting.
I20260929 15:56:28.267232 20714 tablet_bootstrap.cc:492] T a2d9e3c265574aafaadb2fcd22b5adee P feaa4e02253c4e2c883fc46f12806187: No bootstrap required, opened a new log
I20260929 15:56:28.267591 20714 ts_tablet_manager.cc:1397] T a2d9e3c265574aafaadb2fcd22b5adee P feaa4e02253c4e2c883fc46f12806187: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 15:56:28.271445 20716 tablet_bootstrap.cc:654] T a2d9e3c265574aafaadb2fcd22b5adee P d1c2b3a38f26484cb4cf982abf0b1e94: Neither blocks nor log segments found. Creating new log.
I20260929 15:56:28.272939 20716 log.cc:824] T a2d9e3c265574aafaadb2fcd22b5adee P d1c2b3a38f26484cb4cf982abf0b1e94: Log is configured to *not* fsync() on all Append() calls
I20260929 15:56:28.279513 20714 raft_consensus.cc:348] T a2d9e3c265574aafaadb2fcd22b5adee P feaa4e02253c4e2c883fc46f12806187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c2b3a38f26484cb4cf982abf0b1e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36111 } } peers { permanent_uuid: "f9d2efa77e03489cb9b391806e126d71" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43169 } } peers { permanent_uuid: "feaa4e02253c4e2c883fc46f12806187" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37287 } }
I20260929 15:56:28.280311 20714 raft_consensus.cc:374] T a2d9e3c265574aafaadb2fcd22b5adee P feaa4e02253c4e2c883fc46f12806187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:56:28.280447 20714 raft_consensus.cc:729] T a2d9e3c265574aafaadb2fcd22b5adee P feaa4e02253c4e2c883fc46f12806187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feaa4e02253c4e2c883fc46f12806187, State: Initialized, Role: FOLLOWER
I20260929 15:56:28.281229 20714 consensus_queue.cc:260] T a2d9e3c265574aafaadb2fcd22b5adee P feaa4e02253c4e2c883fc46f12806187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c2b3a38f26484cb4cf982abf0b1e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36111 } } peers { permanent_uuid: "f9d2efa77e03489cb9b391806e126d71" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43169 } } peers { permanent_uuid: "feaa4e02253c4e2c883fc46f12806187" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37287 } }
I20260929 15:56:28.282866 20696 heartbeater.cc:503] Master 127.0.97.62:40595 was elected leader, sending a full tablet report...
I20260929 15:56:28.283584 20714 ts_tablet_manager.cc:1428] T a2d9e3c265574aafaadb2fcd22b5adee P feaa4e02253c4e2c883fc46f12806187: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260929 15:56:28.284189 20716 tablet_bootstrap.cc:492] T a2d9e3c265574aafaadb2fcd22b5adee P d1c2b3a38f26484cb4cf982abf0b1e94: No bootstrap required, opened a new log
I20260929 15:56:28.284209 20715 tablet_bootstrap.cc:492] T a2d9e3c265574aafaadb2fcd22b5adee P f9d2efa77e03489cb9b391806e126d71: No bootstrap required, opened a new log
I20260929 15:56:28.284485 20715 ts_tablet_manager.cc:1397] T a2d9e3c265574aafaadb2fcd22b5adee P f9d2efa77e03489cb9b391806e126d71: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.012s
I20260929 15:56:28.284474 20716 ts_tablet_manager.cc:1397] T a2d9e3c265574aafaadb2fcd22b5adee P d1c2b3a38f26484cb4cf982abf0b1e94: Time spent bootstrapping tablet: real 0.019s user 0.015s sys 0.000s
I20260929 15:56:28.292256 20715 raft_consensus.cc:348] T a2d9e3c265574aafaadb2fcd22b5adee P f9d2efa77e03489cb9b391806e126d71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c2b3a38f26484cb4cf982abf0b1e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36111 } } peers { permanent_uuid: "f9d2efa77e03489cb9b391806e126d71" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43169 } } peers { permanent_uuid: "feaa4e02253c4e2c883fc46f12806187" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37287 } }
I20260929 15:56:28.292604 20715 raft_consensus.cc:374] T a2d9e3c265574aafaadb2fcd22b5adee P f9d2efa77e03489cb9b391806e126d71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:56:28.292727 20715 raft_consensus.cc:729] T a2d9e3c265574aafaadb2fcd22b5adee P f9d2efa77e03489cb9b391806e126d71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d2efa77e03489cb9b391806e126d71, State: Initialized, Role: FOLLOWER
I20260929 15:56:28.293079 20716 raft_consensus.cc:348] T a2d9e3c265574aafaadb2fcd22b5adee P d1c2b3a38f26484cb4cf982abf0b1e94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c2b3a38f26484cb4cf982abf0b1e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36111 } } peers { permanent_uuid: "f9d2efa77e03489cb9b391806e126d71" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43169 } } peers { permanent_uuid: "feaa4e02253c4e2c883fc46f12806187" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37287 } }
I20260929 15:56:28.293431 20716 raft_consensus.cc:374] T a2d9e3c265574aafaadb2fcd22b5adee P d1c2b3a38f26484cb4cf982abf0b1e94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:56:28.293516 20716 raft_consensus.cc:729] T a2d9e3c265574aafaadb2fcd22b5adee P d1c2b3a38f26484cb4cf982abf0b1e94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c2b3a38f26484cb4cf982abf0b1e94, State: Initialized, Role: FOLLOWER
I20260929 15:56:28.293298 20715 consensus_queue.cc:260] T a2d9e3c265574aafaadb2fcd22b5adee P f9d2efa77e03489cb9b391806e126d71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c2b3a38f26484cb4cf982abf0b1e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36111 } } peers { permanent_uuid: "f9d2efa77e03489cb9b391806e126d71" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43169 } } peers { permanent_uuid: "feaa4e02253c4e2c883fc46f12806187" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37287 } }
I20260929 15:56:28.294034 20716 consensus_queue.cc:260] T a2d9e3c265574aafaadb2fcd22b5adee P d1c2b3a38f26484cb4cf982abf0b1e94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c2b3a38f26484cb4cf982abf0b1e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36111 } } peers { permanent_uuid: "f9d2efa77e03489cb9b391806e126d71" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43169 } } peers { permanent_uuid: "feaa4e02253c4e2c883fc46f12806187" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37287 } }
I20260929 15:56:28.294412 20435 heartbeater.cc:503] Master 127.0.97.62:40595 was elected leader, sending a full tablet report...
I20260929 15:56:28.295082 20715 ts_tablet_manager.cc:1428] T a2d9e3c265574aafaadb2fcd22b5adee P f9d2efa77e03489cb9b391806e126d71: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260929 15:56:28.295089 20566 heartbeater.cc:503] Master 127.0.97.62:40595 was elected leader, sending a full tablet report...
I20260929 15:56:28.296720 20716 ts_tablet_manager.cc:1428] T a2d9e3c265574aafaadb2fcd22b5adee P d1c2b3a38f26484cb4cf982abf0b1e94: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
W20260929 15:56:28.422467 20697 tablet.cc:2369] T a2d9e3c265574aafaadb2fcd22b5adee P feaa4e02253c4e2c883fc46f12806187: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:56:28.460868 20722 raft_consensus.cc:482] T a2d9e3c265574aafaadb2fcd22b5adee P d1c2b3a38f26484cb4cf982abf0b1e94 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:56:28.461231 20722 raft_consensus.cc:504] T a2d9e3c265574aafaadb2fcd22b5adee P d1c2b3a38f26484cb4cf982abf0b1e94 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c2b3a38f26484cb4cf982abf0b1e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36111 } } peers { permanent_uuid: "f9d2efa77e03489cb9b391806e126d71" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43169 } } peers { permanent_uuid: "feaa4e02253c4e2c883fc46f12806187" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37287 } }
I20260929 15:56:28.463074 20722 leader_election.cc:288] T a2d9e3c265574aafaadb2fcd22b5adee P d1c2b3a38f26484cb4cf982abf0b1e94 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9d2efa77e03489cb9b391806e126d71 (127.0.97.1:43169), feaa4e02253c4e2c883fc46f12806187 (127.0.97.3:37287)
I20260929 15:56:28.470332 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:28.463444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43169 (local address 127.0.97.2:49865)
0929 15:56:28.464154 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:28.464174 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:28.464319 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:28.464646 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:28.464650 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:28.464687 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:56:28.465135 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:28.465146 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:28.466393 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:28.466397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:28.468336 (+ 1939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:28.468351 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:28.469760 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:28.469765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:28.469886 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:28.469911 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 15:56:28.469988 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 15:56:28.470076 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"mutex_wait_us":24,"thread_start_us":349,"threads_started":1}
I20260929 15:56:28.470458 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:28.464138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49865 (local address 127.0.97.1:43169)
0929 15:56:28.464325 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:28.464330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:28.464350 (+ 20us) server_negotiation.cc:407] Connection header received
0929 15:56:28.464434 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:28.464439 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:28.464517 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:28.464625 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:56:28.465329 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:28.466267 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:28.468543 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:28.469569 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:28.469703 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:28.469795 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:28.470052 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 15:56:28.470181 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 15:56:28.471390 20724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:28.463480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37287 (local address 127.0.97.2:46399)
0929 15:56:28.465121 (+ 1641us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:28.465139 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:56:28.465266 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:56:28.465703 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:56:28.465708 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:56:28.465737 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:56:28.466070 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:28.466078 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:28.467127 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:28.467132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:28.468936 (+ 1804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:56:28.468946 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:56:28.470725 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:56:28.470731 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:56:28.470919 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:28.470956 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 15:56:28.471050 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 15:56:28.471151 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1513,"mutex_wait_us":3,"thread_start_us":1304,"threads_started":1}
I20260929 15:56:28.471661 20713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:28.463666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46399 (local address 127.0.97.3:37287)
0929 15:56:28.463891 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:28.463897 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:28.465239 (+ 1342us) server_negotiation.cc:407] Connection header received
0929 15:56:28.465501 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:28.465506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:28.465579 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:28.465684 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:56:28.466216 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:28.466975 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:28.469122 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:28.470359 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:28.470491 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:28.470589 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:28.471127 (+ 538us) server_negotiation.cc:299] Negotiation successful
0929 15:56:28.471265 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 15:56:28.471510 20390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2d9e3c265574aafaadb2fcd22b5adee" candidate_uuid: "d1c2b3a38f26484cb4cf982abf0b1e94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d2efa77e03489cb9b391806e126d71" is_pre_election: true
I20260929 15:56:28.472086 20390 raft_consensus.cc:2454] T a2d9e3c265574aafaadb2fcd22b5adee P f9d2efa77e03489cb9b391806e126d71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c2b3a38f26484cb4cf982abf0b1e94 in term 0.
I20260929 15:56:28.472599 20651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2d9e3c265574aafaadb2fcd22b5adee" candidate_uuid: "d1c2b3a38f26484cb4cf982abf0b1e94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feaa4e02253c4e2c883fc46f12806187" is_pre_election: true
I20260929 15:56:28.472733 20455 leader_election.cc:302] T a2d9e3c265574aafaadb2fcd22b5adee P d1c2b3a38f26484cb4cf982abf0b1e94 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1c2b3a38f26484cb4cf982abf0b1e94, f9d2efa77e03489cb9b391806e126d71; no voters:
I20260929 15:56:28.473080 20651 raft_consensus.cc:2454] T a2d9e3c265574aafaadb2fcd22b5adee P feaa4e02253c4e2c883fc46f12806187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c2b3a38f26484cb4cf982abf0b1e94 in term 0.
I20260929 15:56:28.473167 20722 raft_consensus.cc:2781] T a2d9e3c265574aafaadb2fcd22b5adee P d1c2b3a38f26484cb4cf982abf0b1e94 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:56:28.473335 20722 raft_consensus.cc:482] T a2d9e3c265574aafaadb2fcd22b5adee P d1c2b3a38f26484cb4cf982abf0b1e94 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:56:28.473480 20722 raft_consensus.cc:3037] T a2d9e3c265574aafaadb2fcd22b5adee P d1c2b3a38f26484cb4cf982abf0b1e94 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:56:28.475767 20722 raft_consensus.cc:504] T a2d9e3c265574aafaadb2fcd22b5adee P d1c2b3a38f26484cb4cf982abf0b1e94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c2b3a38f26484cb4cf982abf0b1e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36111 } } peers { permanent_uuid: "f9d2efa77e03489cb9b391806e126d71" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43169 } } peers { permanent_uuid: "feaa4e02253c4e2c883fc46f12806187" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37287 } }
I20260929 15:56:28.476404 20722 leader_election.cc:288] T a2d9e3c265574aafaadb2fcd22b5adee P d1c2b3a38f26484cb4cf982abf0b1e94 [CANDIDATE]: Term 1 election: Requested vote from peers f9d2efa77e03489cb9b391806e126d71 (127.0.97.1:43169), feaa4e02253c4e2c883fc46f12806187 (127.0.97.3:37287)
I20260929 15:56:28.476878 20390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2d9e3c265574aafaadb2fcd22b5adee" candidate_uuid: "d1c2b3a38f26484cb4cf982abf0b1e94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d2efa77e03489cb9b391806e126d71"
I20260929 15:56:28.477108 20390 raft_consensus.cc:3037] T a2d9e3c265574aafaadb2fcd22b5adee P f9d2efa77e03489cb9b391806e126d71 [term 0 FOLLOWER]: Advancing to term 1
W20260929 15:56:28.477222 20567 tablet.cc:2369] T a2d9e3c265574aafaadb2fcd22b5adee P d1c2b3a38f26484cb4cf982abf0b1e94: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:56:28.477185 20651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2d9e3c265574aafaadb2fcd22b5adee" candidate_uuid: "d1c2b3a38f26484cb4cf982abf0b1e94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feaa4e02253c4e2c883fc46f12806187"
I20260929 15:56:28.477591 20651 raft_consensus.cc:3037] T a2d9e3c265574aafaadb2fcd22b5adee P feaa4e02253c4e2c883fc46f12806187 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:56:28.479562 20390 raft_consensus.cc:2454] T a2d9e3c265574aafaadb2fcd22b5adee P f9d2efa77e03489cb9b391806e126d71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c2b3a38f26484cb4cf982abf0b1e94 in term 1.
I20260929 15:56:28.480150 20455 leader_election.cc:302] T a2d9e3c265574aafaadb2fcd22b5adee P d1c2b3a38f26484cb4cf982abf0b1e94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1c2b3a38f26484cb4cf982abf0b1e94, f9d2efa77e03489cb9b391806e126d71; no voters:
I20260929 15:56:28.480459 20722 raft_consensus.cc:2781] T a2d9e3c265574aafaadb2fcd22b5adee P d1c2b3a38f26484cb4cf982abf0b1e94 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:56:28.480721 20651 raft_consensus.cc:2454] T a2d9e3c265574aafaadb2fcd22b5adee P feaa4e02253c4e2c883fc46f12806187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c2b3a38f26484cb4cf982abf0b1e94 in term 1.
I20260929 15:56:28.481010 20722 raft_consensus.cc:686] T a2d9e3c265574aafaadb2fcd22b5adee P d1c2b3a38f26484cb4cf982abf0b1e94 [term 1 LEADER]: Becoming Leader. State: Replica: d1c2b3a38f26484cb4cf982abf0b1e94, State: Running, Role: LEADER
I20260929 15:56:28.481559 20722 consensus_queue.cc:237] T a2d9e3c265574aafaadb2fcd22b5adee P d1c2b3a38f26484cb4cf982abf0b1e94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c2b3a38f26484cb4cf982abf0b1e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36111 } } peers { permanent_uuid: "f9d2efa77e03489cb9b391806e126d71" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43169 } } peers { permanent_uuid: "feaa4e02253c4e2c883fc46f12806187" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37287 } }
I20260929 15:56:28.486254 20249 catalog_manager.cc:5495] T a2d9e3c265574aafaadb2fcd22b5adee P d1c2b3a38f26484cb4cf982abf0b1e94 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1c2b3a38f26484cb4cf982abf0b1e94 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "d1c2b3a38f26484cb4cf982abf0b1e94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c2b3a38f26484cb4cf982abf0b1e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9d2efa77e03489cb9b391806e126d71" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feaa4e02253c4e2c883fc46f12806187" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37287 } health_report { overall_health: UNKNOWN } } }
W20260929 15:56:28.520591 20436 tablet.cc:2369] T a2d9e3c265574aafaadb2fcd22b5adee P f9d2efa77e03489cb9b391806e126d71: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:56:28.535344 20709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:28.529883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43390 (local address 127.0.97.2:36111)
0929 15:56:28.530116 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:28.530122 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:28.530307 (+ 185us) server_negotiation.cc:407] Connection header received
0929 15:56:28.530556 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:28.530560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:28.530637 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:28.530800 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:56:28.531165 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:28.531882 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:28.533209 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:28.533702 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:28.533839 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:28.534301 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:56:28.534794 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:56:28.534959 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:28.535015 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 15:56:28.535092 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 15:56:28.543613 20651 raft_consensus.cc:1261] T a2d9e3c265574aafaadb2fcd22b5adee P feaa4e02253c4e2c883fc46f12806187 [term 1 FOLLOWER]: Refusing update from remote peer d1c2b3a38f26484cb4cf982abf0b1e94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:56:28.543874 20390 raft_consensus.cc:1261] T a2d9e3c265574aafaadb2fcd22b5adee P f9d2efa77e03489cb9b391806e126d71 [term 1 FOLLOWER]: Refusing update from remote peer d1c2b3a38f26484cb4cf982abf0b1e94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:56:28.544741 20725 consensus_queue.cc:1035] T a2d9e3c265574aafaadb2fcd22b5adee P d1c2b3a38f26484cb4cf982abf0b1e94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feaa4e02253c4e2c883fc46f12806187" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:56:28.545120 20722 consensus_queue.cc:1035] T a2d9e3c265574aafaadb2fcd22b5adee P d1c2b3a38f26484cb4cf982abf0b1e94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d2efa77e03489cb9b391806e126d71" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:56:28.557281 20728 mvcc.cc:205] Tried to move back new op lower bound from 7334696503463731200 to 7334696503224258560. Current Snapshot: MvccSnapshot[applied={T|T < 7334696503463731200}]
I20260929 15:56:28.559142 20730 mvcc.cc:205] Tried to move back new op lower bound from 7334696503463731200 to 7334696503224258560. Current Snapshot: MvccSnapshot[applied={T|T < 7334696503463731200}]
I20260929 15:56:28.561545 20729 mvcc.cc:205] Tried to move back new op lower bound from 7334696503463731200 to 7334696503224258560. Current Snapshot: MvccSnapshot[applied={T|T < 7334696503463731200}]
I20260929 15:56:28.570192 20249 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:56:28.570420 20249 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:56:28.574244 20249 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 619707b8e55042a29116bc9486a86f99: Sending DeleteTablet for 3 replicas of tablet a2d9e3c265574aafaadb2fcd22b5adee
I20260929 15:56:28.575382 20501 tablet_service.cc:1514] Processing DeleteTablet for tablet a2d9e3c265574aafaadb2fcd22b5adee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:56:28 UTC) from {username='slave'} at 127.0.0.1:43384
I20260929 15:56:28.575443 20370 tablet_service.cc:1514] Processing DeleteTablet for tablet a2d9e3c265574aafaadb2fcd22b5adee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:56:28 UTC) from {username='slave'} at 127.0.0.1:58776
I20260929 15:56:28.575703 20631 tablet_service.cc:1514] Processing DeleteTablet for tablet a2d9e3c265574aafaadb2fcd22b5adee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:56:28 UTC) from {username='slave'} at 127.0.0.1:58328
I20260929 15:56:28.581095 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 20308
I20260929 15:56:28.592099 20739 tablet_replica.cc:330] T a2d9e3c265574aafaadb2fcd22b5adee P feaa4e02253c4e2c883fc46f12806187: stopping tablet replica
I20260929 15:56:28.592295 20738 tablet_replica.cc:330] T a2d9e3c265574aafaadb2fcd22b5adee P f9d2efa77e03489cb9b391806e126d71: stopping tablet replica
I20260929 15:56:28.592581 20739 raft_consensus.cc:2229] T a2d9e3c265574aafaadb2fcd22b5adee P feaa4e02253c4e2c883fc46f12806187 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:56:28.593109 20737 tablet_replica.cc:330] T a2d9e3c265574aafaadb2fcd22b5adee P d1c2b3a38f26484cb4cf982abf0b1e94: stopping tablet replica
I20260929 15:56:28.593096 20739 raft_consensus.cc:2258] T a2d9e3c265574aafaadb2fcd22b5adee P feaa4e02253c4e2c883fc46f12806187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:56:28.593508 20737 raft_consensus.cc:2229] T a2d9e3c265574aafaadb2fcd22b5adee P d1c2b3a38f26484cb4cf982abf0b1e94 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:56:28.594600 20737 raft_consensus.cc:2258] T a2d9e3c265574aafaadb2fcd22b5adee P d1c2b3a38f26484cb4cf982abf0b1e94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:56:28.594944 20739 ts_tablet_manager.cc:1905] T a2d9e3c265574aafaadb2fcd22b5adee P feaa4e02253c4e2c883fc46f12806187: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:56:28.595129 20738 raft_consensus.cc:2229] T a2d9e3c265574aafaadb2fcd22b5adee P f9d2efa77e03489cb9b391806e126d71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:56:28.595614 20738 raft_consensus.cc:2258] T a2d9e3c265574aafaadb2fcd22b5adee P f9d2efa77e03489cb9b391806e126d71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:56:28.596472 20737 ts_tablet_manager.cc:1905] T a2d9e3c265574aafaadb2fcd22b5adee P d1c2b3a38f26484cb4cf982abf0b1e94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:56:28.597608 20738 ts_tablet_manager.cc:1905] T a2d9e3c265574aafaadb2fcd22b5adee P f9d2efa77e03489cb9b391806e126d71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:56:28.601419 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:28.582653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58788 (local address 127.0.97.1:43169)
0929 15:56:28.582846 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:28.582852 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:28.584364 (+ 1512us) server_negotiation.cc:407] Connection header received
0929 15:56:28.585214 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:28.585224 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:28.585399 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:28.585495 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:28.586834 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:28.588372 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:28.592133 (+ 3761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:28.592819 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:28.592913 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:28.599657 (+ 6744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:28.599671 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:28.599689 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:28.600445 (+ 756us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:28.600506 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:28.600531 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:28.600617 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:28.600941 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 15:56:28.601057 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 15:56:28.602106 20739 ts_tablet_manager.cc:1918] T a2d9e3c265574aafaadb2fcd22b5adee P feaa4e02253c4e2c883fc46f12806187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:56:28.602280 20739 log.cc:1196] T a2d9e3c265574aafaadb2fcd22b5adee P feaa4e02253c4e2c883fc46f12806187: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-2/wal/wals/a2d9e3c265574aafaadb2fcd22b5adee
I20260929 15:56:28.602852 20739 ts_tablet_manager.cc:1939] T a2d9e3c265574aafaadb2fcd22b5adee P feaa4e02253c4e2c883fc46f12806187: Deleting consensus metadata
I20260929 15:56:28.602871 20430 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:56:28.603925 20737 ts_tablet_manager.cc:1918] T a2d9e3c265574aafaadb2fcd22b5adee P d1c2b3a38f26484cb4cf982abf0b1e94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:56:28.604110 20737 log.cc:1196] T a2d9e3c265574aafaadb2fcd22b5adee P d1c2b3a38f26484cb4cf982abf0b1e94: Deleting WAL directory at /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697075657932-388-0/minicluster-data/ts-1/wal/wals/a2d9e3c265574aafaadb2fcd22b5adee
I20260929 15:56:28.604660 20737 ts_tablet_manager.cc:1939] T a2d9e3c265574aafaadb2fcd22b5adee P d1c2b3a38f26484cb4cf982abf0b1e94: Deleting consensus metadata
I20260929 15:56:28.605167 20235 catalog_manager.cc:4841] TS feaa4e02253c4e2c883fc46f12806187 (127.0.97.3:37287): tablet a2d9e3c265574aafaadb2fcd22b5adee (table test-table [id=2ca5f044775149799114ec4451c8ce05]) successfully deleted
I20260929 15:56:28.606421 20233 catalog_manager.cc:4841] TS d1c2b3a38f26484cb4cf982abf0b1e94 (127.0.97.2:36111): tablet a2d9e3c265574aafaadb2fcd22b5adee (table test-table [id=2ca5f044775149799114ec4451c8ce05]) successfully deleted
I20260929 15:56:28.697642 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 20308
W20260929 15:56:28.720983 20234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:43169 (error 108)
I20260929 15:56:28.722028 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 20439
I20260929 15:56:28.732604 20709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:28.723946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43400 (local address 127.0.97.2:36111)
0929 15:56:28.724129 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:28.724135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:28.724155 (+ 20us) server_negotiation.cc:407] Connection header received
0929 15:56:28.724215 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:28.724220 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:28.724329 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:28.724435 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:28.725093 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:28.726426 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:28.728285 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:28.728961 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:28.729078 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:28.730584 (+ 1506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:28.730593 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:28.730609 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:28.731363 (+ 754us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:28.731421 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:28.731444 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:28.731546 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:28.732113 (+ 567us) server_negotiation.cc:299] Negotiation successful
0929 15:56:28.732215 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 15:56:28.733383 20561 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:56:28.734292 20708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:28.733668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43169 (local address 127.0.0.1:58790)
0929 15:56:28.733901 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:28.734076 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 15:56:28.734625 20234 catalog_manager.cc:4584] TS f9d2efa77e03489cb9b391806e126d71 (127.0.97.1:43169): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a2d9e3c265574aafaadb2fcd22b5adee: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43169: connect: Connection refused (error 111)
I20260929 15:56:28.788316 20708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:28.787850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43169 (local address 127.0.0.1:58798)
0929 15:56:28.788053 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:28.788147 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 15:56:28.789268 20708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:28.788935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43169 (local address 127.0.0.1:58806)
0929 15:56:28.789075 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:28.789154 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 15:56:28.847175 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 20439
I20260929 15:56:28.849711 20708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:28.849246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43169 (local address 127.0.0.1:58810)
0929 15:56:28.849470 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:28.849556 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 15:56:28.850631 20708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:28.850292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43169 (local address 127.0.0.1:58824)
0929 15:56:28.850438 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:28.850522 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 15:56:28.866364 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 20569
I20260929 15:56:28.873080 20713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:28.867354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58338 (local address 127.0.97.3:37287)
0929 15:56:28.867561 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:28.867566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:28.867584 (+ 18us) server_negotiation.cc:407] Connection header received
0929 15:56:28.867658 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:28.867662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:28.867748 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:28.867843 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:28.868352 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:28.869057 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:28.870413 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:28.870789 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:28.870916 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:28.871881 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:28.871889 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:28.871902 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:28.872451 (+ 549us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:28.872496 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:28.872515 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:28.872609 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:28.872748 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 15:56:28.872830 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 15:56:28.873739 20691 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:56:28.931416 20708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:28.930894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43169 (local address 127.0.0.1:58826)
0929 15:56:28.931120 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:28.931222 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 15:56:28.932955 20708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:28.932353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43169 (local address 127.0.0.1:58832)
0929 15:56:28.932744 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0929 15:56:28.932817 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276}
I20260929 15:56:28.962167 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 20569
I20260929 15:56:28.978984 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 20218
I20260929 15:56:28.980139 20279 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:56:29.089030 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 20218
2026-09-29T15:56:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2895 ms)
[----------] 6 tests from EncryptionPolicyTest (21790 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:56:29 dist-test-slave-r0ck krb5kdc[20748](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:56:29 dist-test-slave-r0ck krb5kdc[20748](info): set up 2 sockets
Sep 29 15:56:29 dist-test-slave-r0ck krb5kdc[20748](info): commencing operation
krb5kdc: starting...
W20260929 15:56:30.205816 388 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:56:30 dist-test-slave-r0ck krb5kdc[20748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:56:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:56:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 15:56:30.485263 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38113
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:56:30.820577 20761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:56:30.820861 20761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:56:30.820959 20761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:56:30.828042 20761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:56:30.828154 20761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:56:30.828203 20761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:56:30.828263 20761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:56:30.828307 20761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:56:30.837890 20761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38113
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:56:30.838999 20761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:56:30.840181 20761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:56:30.852229 20766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:30.852193 20767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:30.852277 20769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:56:30.853080 20761 server_base.cc:1034] running on GCE node
I20260929 15:56:30.853945 20761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:56:30.855399 20761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:56:30.856662 20761 hybrid_clock.cc:648] HybridClock initialized: now 1790697390856606 us; error 45 us; skew 500 ppm
I20260929 15:56:30.857049 20761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:56:30 dist-test-slave-r0ck krb5kdc[20748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:56:30.861462 20761 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 15:56:30.863221 20761 webserver.cc:460] Webserver started at http://127.0.0.1:39121/ using document root <none> and password file <none>
I20260929 15:56:30.863937 20761 fs_manager.cc:359] Metadata directory not provided
I20260929 15:56:30.864084 20761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:56:30.864416 20761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:56:30.867512 20761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "ed3e488b984a44469ffa7c8fd86e953c"
format_stamp: "Formatted at 2026-09-29 15:56:30 on dist-test-slave-r0ck"
server_key: "9ad96870dda5ec7c68ea0bf30bc8fc65"
server_key_iv: "05e861c9f213658699684708f9b76616"
server_key_version: "encryptionkey@0"
I20260929 15:56:30.868325 20761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "ed3e488b984a44469ffa7c8fd86e953c"
format_stamp: "Formatted at 2026-09-29 15:56:30 on dist-test-slave-r0ck"
server_key: "9ad96870dda5ec7c68ea0bf30bc8fc65"
server_key_iv: "05e861c9f213658699684708f9b76616"
server_key_version: "encryptionkey@0"
I20260929 15:56:30.873836 20761 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260929 15:56:30.877926 20776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:30.879427 20761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 15:56:30.879635 20761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "ed3e488b984a44469ffa7c8fd86e953c"
format_stamp: "Formatted at 2026-09-29 15:56:30 on dist-test-slave-r0ck"
server_key: "9ad96870dda5ec7c68ea0bf30bc8fc65"
server_key_iv: "05e861c9f213658699684708f9b76616"
server_key_version: "encryptionkey@0"
I20260929 15:56:30.879846 20761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:56:30.904011 20761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:56:30.908489 20761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:56:30.908777 20761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:56:30.927819 20761 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38113
I20260929 15:56:30.927842 20827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38113 every 8 connection(s)
I20260929 15:56:30.929143 20761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:56:30.933293 20828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:56:30.937270 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 20761
I20260929 15:56:30.937563 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:56:30.938237 388 external_mini_cluster.cc:1428] Setting key b0f3425af78fc65642c021d921e2d64f
I20260929 15:56:30.944268 20828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed3e488b984a44469ffa7c8fd86e953c: Bootstrap starting.
I20260929 15:56:30.947340 20828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed3e488b984a44469ffa7c8fd86e953c: Neither blocks nor log segments found. Creating new log.
I20260929 15:56:30.948459 20828 log.cc:824] T 00000000000000000000000000000000 P ed3e488b984a44469ffa7c8fd86e953c: Log is configured to *not* fsync() on all Append() calls
I20260929 15:56:30.951794 20828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed3e488b984a44469ffa7c8fd86e953c: No bootstrap required, opened a new log
Sep 29 15:56:30 dist-test-slave-r0ck krb5kdc[20748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 15:56:30.962167 20828 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed3e488b984a44469ffa7c8fd86e953c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed3e488b984a44469ffa7c8fd86e953c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38113 } }
I20260929 15:56:30.962626 20828 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed3e488b984a44469ffa7c8fd86e953c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:56:30.962730 20828 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed3e488b984a44469ffa7c8fd86e953c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed3e488b984a44469ffa7c8fd86e953c, State: Initialized, Role: FOLLOWER
I20260929 15:56:30.963346 20831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:30.940301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40113 (local address 127.0.0.1:38113)
0929 15:56:30.941118 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:30.941130 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:30.941174 (+ 44us) server_negotiation.cc:407] Connection header received
0929 15:56:30.943150 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:30.943171 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:30.943569 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:30.943966 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:30.945583 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:30.947042 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:30.948405 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:30.949362 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:30.949563 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:30.954326 (+ 4763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:30.954336 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:30.954359 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:56:30.954421 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:30.958500 (+ 4079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:30.959281 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:30.959289 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:30.959298 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:30.959496 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:30.960035 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:30.960045 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:30.960049 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:30.960791 (+ 742us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:30.961101 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:30.961656 (+ 555us) server_negotiation.cc:299] Negotiation successful
0929 15:56:30.962003 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":366,"threads_started":1}
I20260929 15:56:30.963395 20828 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed3e488b984a44469ffa7c8fd86e953c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed3e488b984a44469ffa7c8fd86e953c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38113 } }
I20260929 15:56:30.963667 20828 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed3e488b984a44469ffa7c8fd86e953c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:56:30.963812 20828 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed3e488b984a44469ffa7c8fd86e953c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:56:30.963994 20828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed3e488b984a44469ffa7c8fd86e953c [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:56:30.966763 20828 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed3e488b984a44469ffa7c8fd86e953c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed3e488b984a44469ffa7c8fd86e953c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38113 } }
I20260929 15:56:30.967270 20828 leader_election.cc:302] T 00000000000000000000000000000000 P ed3e488b984a44469ffa7c8fd86e953c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed3e488b984a44469ffa7c8fd86e953c; no voters:
I20260929 15:56:30.968052 20828 leader_election.cc:288] T 00000000000000000000000000000000 P ed3e488b984a44469ffa7c8fd86e953c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:56:30.968178 20833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed3e488b984a44469ffa7c8fd86e953c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:56:30.968370 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:56:30.969617 20833 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed3e488b984a44469ffa7c8fd86e953c [term 1 LEADER]: Becoming Leader. State: Replica: ed3e488b984a44469ffa7c8fd86e953c, State: Running, Role: LEADER
I20260929 15:56:30.970196 20833 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed3e488b984a44469ffa7c8fd86e953c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed3e488b984a44469ffa7c8fd86e953c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38113 } }
I20260929 15:56:30.970468 20828 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed3e488b984a44469ffa7c8fd86e953c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:56:30.975355 20834 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed3e488b984a44469ffa7c8fd86e953c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed3e488b984a44469ffa7c8fd86e953c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed3e488b984a44469ffa7c8fd86e953c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38113 } } }
I20260929 15:56:30.975791 20834 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed3e488b984a44469ffa7c8fd86e953c [sys.catalog]: This master's current role is: LEADER
I20260929 15:56:30.975389 20835 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed3e488b984a44469ffa7c8fd86e953c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed3e488b984a44469ffa7c8fd86e953c. Latest consensus state: current_term: 1 leader_uuid: "ed3e488b984a44469ffa7c8fd86e953c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed3e488b984a44469ffa7c8fd86e953c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38113 } } }
I20260929 15:56:30.976598 20835 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed3e488b984a44469ffa7c8fd86e953c [sys.catalog]: This master's current role is: LEADER
I20260929 15:56:30.985044 20844 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:56:30.991045 20844 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:56:30.999475 20831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:30.975634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50515 (local address 127.0.0.1:38113)
0929 15:56:30.975850 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:30.975860 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:30.976460 (+ 600us) server_negotiation.cc:407] Connection header received
0929 15:56:30.976815 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:30.976823 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:30.976924 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:30.977010 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:30.979554 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:30.981105 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:30.984220 (+ 3115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:30.985120 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:30.985240 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:30.988542 (+ 3302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:30.988552 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:30.988560 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:56:30.988608 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:30.993744 (+ 5136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:30.995037 (+ 1293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:30.995046 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:30.995051 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:30.995227 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:30.995965 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:30.995973 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:30.995979 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:30.996531 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:30.998451 (+ 1920us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:30.998688 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 15:56:30.998921 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 15:56:31.004282 20844 catalog_manager.cc:1347] Generated new cluster ID: 21d887e8af784da39404774ae69af641
I20260929 15:56:31.004464 20844 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:56:31.052583 20844 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:56:31.053982 20844 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:56:31.063593 20844 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed3e488b984a44469ffa7c8fd86e953c: Generated new TSK 0
I20260929 15:56:31.064369 20844 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:56:31.072499 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 20761
I20260929 15:56:31.073645 20823 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:56:31.168402 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 20761
W20260929 15:56:31.185577 20746 connection.cc:535] client connection to 127.0.0.1:38113 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38113): Connection reset by peer (error 104)
2026-09-29T15:56:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2072 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:56:31 dist-test-slave-r0ck krb5kdc[20860](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:56:31 dist-test-slave-r0ck krb5kdc[20860](info): set up 2 sockets
Sep 29 15:56:31 dist-test-slave-r0ck krb5kdc[20860](info): commencing operation
krb5kdc: starting...
W20260929 15:56:32.273715 388 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:56:32 dist-test-slave-r0ck krb5kdc[20860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:56:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:56:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 15:56:32.530975 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41861
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:56:32.857108 20873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:56:32.857474 20873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:56:32.857647 20873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:56:32.864867 20873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:56:32.865025 20873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:56:32.865134 20873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:56:32.865260 20873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:56:32.865365 20873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:56:32.875109 20873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41861
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:56:32.876437 20873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:56:32.877730 20873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:56:32.889817 20881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:32.890208 20878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:32.890268 20879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:56:32.890833 20873 server_base.cc:1034] running on GCE node
I20260929 15:56:32.891671 20873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:56:32.893136 20873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:56:32.894371 20873 hybrid_clock.cc:648] HybridClock initialized: now 1790697392894317 us; error 52 us; skew 500 ppm
I20260929 15:56:32.894723 20873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:56:32 dist-test-slave-r0ck krb5kdc[20860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:56:32.897794 20873 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 15:56:32.899242 20873 webserver.cc:460] Webserver started at http://127.0.0.1:37569/ using document root <none> and password file <none>
I20260929 15:56:32.899909 20873 fs_manager.cc:359] Metadata directory not provided
I20260929 15:56:32.900055 20873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:56:32.900383 20873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:56:32.903331 20873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "72eb9c5ed99446b3b1dc1057a21c9074"
format_stamp: "Formatted at 2026-09-29 15:56:32 on dist-test-slave-r0ck"
server_key: "76f1df61b65771d3bfa7f3c687070a80"
server_key_iv: "fcc8b253851d8ce6dd78b8db727c12d7"
server_key_version: "encryptionkey@0"
I20260929 15:56:32.904155 20873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "72eb9c5ed99446b3b1dc1057a21c9074"
format_stamp: "Formatted at 2026-09-29 15:56:32 on dist-test-slave-r0ck"
server_key: "76f1df61b65771d3bfa7f3c687070a80"
server_key_iv: "fcc8b253851d8ce6dd78b8db727c12d7"
server_key_version: "encryptionkey@0"
I20260929 15:56:32.909482 20873 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 15:56:32.912971 20888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:32.914515 20873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 15:56:32.914701 20873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "72eb9c5ed99446b3b1dc1057a21c9074"
format_stamp: "Formatted at 2026-09-29 15:56:32 on dist-test-slave-r0ck"
server_key: "76f1df61b65771d3bfa7f3c687070a80"
server_key_iv: "fcc8b253851d8ce6dd78b8db727c12d7"
server_key_version: "encryptionkey@0"
I20260929 15:56:32.914899 20873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:56:32.930063 20873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:56:32.934329 20873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:56:32.934679 20873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:56:32.953285 20873 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41861
I20260929 15:56:32.953331 20939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41861 every 8 connection(s)
I20260929 15:56:32.954631 20873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:56:32.955232 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 20873
I20260929 15:56:32.955463 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:56:32.956082 388 external_mini_cluster.cc:1428] Setting key 5cdbf54b9c7d5bf9958dd9ecad2d20aa
I20260929 15:56:32.958876 20940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:56:32.969425 20943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:32.957924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48161 (local address 127.0.0.1:41861)
0929 15:56:32.958696 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:32.958706 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:32.958752 (+ 46us) server_negotiation.cc:407] Connection header received
0929 15:56:32.960380 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:32.960399 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:32.960770 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:32.961078 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:32.961660 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:32.962972 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:32.964791 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:32.965539 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:32.965687 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:32.966985 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:32.966991 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:32.967005 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:32.967554 (+ 549us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:32.967629 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:32.967653 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:32.967794 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:32.968180 (+ 386us) server_negotiation.cc:299] Negotiation successful
0929 15:56:32.968378 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":366,"threads_started":1}
I20260929 15:56:32.970836 20940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72eb9c5ed99446b3b1dc1057a21c9074: Bootstrap starting.
I20260929 15:56:32.973412 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:56:32.974311 20940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72eb9c5ed99446b3b1dc1057a21c9074: Neither blocks nor log segments found. Creating new log.
I20260929 15:56:32.975462 20940 log.cc:824] T 00000000000000000000000000000000 P 72eb9c5ed99446b3b1dc1057a21c9074: Log is configured to *not* fsync() on all Append() calls
I20260929 15:56:32.978425 20940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72eb9c5ed99446b3b1dc1057a21c9074: No bootstrap required, opened a new log
I20260929 15:56:32.985081 20943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:32.978646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57769 (local address 127.0.0.1:41861)
0929 15:56:32.978860 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:32.978866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:32.979127 (+ 261us) server_negotiation.cc:407] Connection header received
0929 15:56:32.979384 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:32.979390 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:32.979458 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:32.979571 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:32.980096 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:32.980885 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:32.982340 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:32.982831 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:32.982990 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:32.984222 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:32.984226 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:32.984230 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:32.984248 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:32.984288 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:32.984303 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:32.984447 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:32.984645 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 15:56:32.984699 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 15:56:32.987229 20940 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72eb9c5ed99446b3b1dc1057a21c9074 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72eb9c5ed99446b3b1dc1057a21c9074" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41861 } }
I20260929 15:56:32.987604 20940 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72eb9c5ed99446b3b1dc1057a21c9074 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:56:32.987709 20940 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72eb9c5ed99446b3b1dc1057a21c9074 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72eb9c5ed99446b3b1dc1057a21c9074, State: Initialized, Role: FOLLOWER
I20260929 15:56:32.988368 20940 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72eb9c5ed99446b3b1dc1057a21c9074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72eb9c5ed99446b3b1dc1057a21c9074" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41861 } }
I20260929 15:56:32.988608 20940 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72eb9c5ed99446b3b1dc1057a21c9074 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:56:32.988775 20940 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72eb9c5ed99446b3b1dc1057a21c9074 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:56:32.988945 20940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72eb9c5ed99446b3b1dc1057a21c9074 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:56:32.991184 20940 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72eb9c5ed99446b3b1dc1057a21c9074 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72eb9c5ed99446b3b1dc1057a21c9074" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41861 } }
I20260929 15:56:32.991686 20940 leader_election.cc:302] T 00000000000000000000000000000000 P 72eb9c5ed99446b3b1dc1057a21c9074 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72eb9c5ed99446b3b1dc1057a21c9074; no voters:
I20260929 15:56:32.992311 20940 leader_election.cc:288] T 00000000000000000000000000000000 P 72eb9c5ed99446b3b1dc1057a21c9074 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:56:32.992477 20950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72eb9c5ed99446b3b1dc1057a21c9074 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:56:32.993742 20950 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72eb9c5ed99446b3b1dc1057a21c9074 [term 1 LEADER]: Becoming Leader. State: Replica: 72eb9c5ed99446b3b1dc1057a21c9074, State: Running, Role: LEADER
I20260929 15:56:32.994256 20950 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72eb9c5ed99446b3b1dc1057a21c9074 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72eb9c5ed99446b3b1dc1057a21c9074" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41861 } }
I20260929 15:56:32.994531 20940 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72eb9c5ed99446b3b1dc1057a21c9074 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:56:32.997452 20952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72eb9c5ed99446b3b1dc1057a21c9074 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72eb9c5ed99446b3b1dc1057a21c9074. Latest consensus state: current_term: 1 leader_uuid: "72eb9c5ed99446b3b1dc1057a21c9074" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72eb9c5ed99446b3b1dc1057a21c9074" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41861 } } }
I20260929 15:56:32.997902 20952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72eb9c5ed99446b3b1dc1057a21c9074 [sys.catalog]: This master's current role is: LEADER
I20260929 15:56:32.997478 20951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72eb9c5ed99446b3b1dc1057a21c9074 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72eb9c5ed99446b3b1dc1057a21c9074" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72eb9c5ed99446b3b1dc1057a21c9074" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41861 } } }
I20260929 15:56:32.998365 20951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72eb9c5ed99446b3b1dc1057a21c9074 [sys.catalog]: This master's current role is: LEADER
I20260929 15:56:33.008159 20958 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:56:33.015988 20958 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:56:33.027629 20958 catalog_manager.cc:1347] Generated new cluster ID: f1734b955e604aff813470a5002d25a4
I20260929 15:56:33.027731 20958 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:56:33.052433 20958 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:56:33.053828 20958 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:56:33.063257 20958 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72eb9c5ed99446b3b1dc1057a21c9074: Generated new TSK 0
I20260929 15:56:33.064122 20958 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:56:33.079138 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 20873
W20260929 15:56:33.080380 20935 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48161
W20260929 15:56:33.081151 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 20873: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:56:33.081377 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 20873
W20260929 15:56:33.099144 20858 connection.cc:535] client connection to 127.0.0.1:41861 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41861): Connection reset by peer (error 104)
2026-09-29T15:56:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1913 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:56:33 dist-test-slave-r0ck krb5kdc[20971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:56:33 dist-test-slave-r0ck krb5kdc[20971](info): set up 2 sockets
Sep 29 15:56:33 dist-test-slave-r0ck krb5kdc[20971](info): commencing operation
krb5kdc: starting...
W20260929 15:56:34.205806 388 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:56:34 dist-test-slave-r0ck krb5kdc[20971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:56:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:56:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 15:56:34.457413 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46667
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:56:34.784900 20984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:56:34.785166 20984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:56:34.785255 20984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:56:34.792307 20984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:56:34.792423 20984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:56:34.792469 20984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:56:34.792531 20984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:56:34.792575 20984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:56:34.802356 20984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46667
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:56:34.803532 20984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:56:34.804701 20984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:56:34.816349 20989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:34.816320 20990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:34.816408 20992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:56:34.817145 20984 server_base.cc:1034] running on GCE node
I20260929 15:56:34.818017 20984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:56:34.819458 20984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:56:34.820677 20984 hybrid_clock.cc:648] HybridClock initialized: now 1790697394820618 us; error 45 us; skew 500 ppm
I20260929 15:56:34.821039 20984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:56:34 dist-test-slave-r0ck krb5kdc[20971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:56:34.827461 20984 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 15:56:34.828960 20984 webserver.cc:460] Webserver started at http://127.0.0.1:35811/ using document root <none> and password file <none>
I20260929 15:56:34.829718 20984 fs_manager.cc:359] Metadata directory not provided
I20260929 15:56:34.829849 20984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:56:34.830160 20984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:56:34.833124 20984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "75910e054c2d4704b6961aec0c9fcd4b"
format_stamp: "Formatted at 2026-09-29 15:56:34 on dist-test-slave-r0ck"
server_key: "19bce1a6378b5f2e082e9fcb7ae5c069"
server_key_iv: "a2816fa0311e4a504b313b9d59701e4a"
server_key_version: "encryptionkey@0"
I20260929 15:56:34.833935 20984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "75910e054c2d4704b6961aec0c9fcd4b"
format_stamp: "Formatted at 2026-09-29 15:56:34 on dist-test-slave-r0ck"
server_key: "19bce1a6378b5f2e082e9fcb7ae5c069"
server_key_iv: "a2816fa0311e4a504b313b9d59701e4a"
server_key_version: "encryptionkey@0"
I20260929 15:56:34.838788 20984 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 15:56:34.842150 20999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:34.843396 20984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:56:34.843581 20984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "75910e054c2d4704b6961aec0c9fcd4b"
format_stamp: "Formatted at 2026-09-29 15:56:34 on dist-test-slave-r0ck"
server_key: "19bce1a6378b5f2e082e9fcb7ae5c069"
server_key_iv: "a2816fa0311e4a504b313b9d59701e4a"
server_key_version: "encryptionkey@0"
I20260929 15:56:34.843781 20984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:56:34.849406 20984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:56:34.853216 20984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:56:34.853554 20984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:56:34.872747 20984 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46667
I20260929 15:56:34.872751 21050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46667 every 8 connection(s)
I20260929 15:56:34.874208 20984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:56:34.877864 21051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:56:34.883051 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 20984
I20260929 15:56:34.883244 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:56:34.883682 388 external_mini_cluster.cc:1428] Setting key 3396cb8c1da175042204b5e150cfea43
I20260929 15:56:34.889585 21051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75910e054c2d4704b6961aec0c9fcd4b: Bootstrap starting.
I20260929 15:56:34.892503 21054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:34.885500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47223 (local address 127.0.0.1:46667)
0929 15:56:34.886344 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:34.886362 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:34.886412 (+ 50us) server_negotiation.cc:407] Connection header received
0929 15:56:34.888820 (+ 2408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:34.888846 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:34.889266 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:34.889715 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:34.890637 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:34.890655 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:34.890672 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:34.890692 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:34.890762 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:34.890789 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:34.891002 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:34.891135 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 15:56:34.891342 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":332,"threads_started":1}
I20260929 15:56:34.893513 21051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75910e054c2d4704b6961aec0c9fcd4b: Neither blocks nor log segments found. Creating new log.
I20260929 15:56:34.894575 21051 log.cc:824] T 00000000000000000000000000000000 P 75910e054c2d4704b6961aec0c9fcd4b: Log is configured to *not* fsync() on all Append() calls
I20260929 15:56:34.896296 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:56:34.897543 21051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75910e054c2d4704b6961aec0c9fcd4b: No bootstrap required, opened a new log
I20260929 15:56:34.902894 21054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:34.900560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36481 (local address 127.0.0.1:46667)
0929 15:56:34.900762 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:34.900770 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:34.901015 (+ 245us) server_negotiation.cc:407] Connection header received
0929 15:56:34.901157 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:34.901163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:34.901220 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:34.901327 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:34.902261 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:34.902266 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:34.902271 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:34.902289 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:34.902336 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:34.902357 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:34.902502 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:34.902543 (+ 41us) server_negotiation.cc:299] Negotiation successful
0929 15:56:34.902600 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 15:56:34.905884 21051 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75910e054c2d4704b6961aec0c9fcd4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75910e054c2d4704b6961aec0c9fcd4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46667 } }
I20260929 15:56:34.906245 21051 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75910e054c2d4704b6961aec0c9fcd4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:56:34.906359 21051 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75910e054c2d4704b6961aec0c9fcd4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75910e054c2d4704b6961aec0c9fcd4b, State: Initialized, Role: FOLLOWER
I20260929 15:56:34.906898 21051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75910e054c2d4704b6961aec0c9fcd4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75910e054c2d4704b6961aec0c9fcd4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46667 } }
I20260929 15:56:34.907127 21051 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75910e054c2d4704b6961aec0c9fcd4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:56:34.907274 21051 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75910e054c2d4704b6961aec0c9fcd4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:56:34.907442 21051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75910e054c2d4704b6961aec0c9fcd4b [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:56:34.909837 21051 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75910e054c2d4704b6961aec0c9fcd4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75910e054c2d4704b6961aec0c9fcd4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46667 } }
I20260929 15:56:34.910315 21051 leader_election.cc:302] T 00000000000000000000000000000000 P 75910e054c2d4704b6961aec0c9fcd4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75910e054c2d4704b6961aec0c9fcd4b; no voters:
I20260929 15:56:34.910934 21051 leader_election.cc:288] T 00000000000000000000000000000000 P 75910e054c2d4704b6961aec0c9fcd4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:56:34.911105 21061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75910e054c2d4704b6961aec0c9fcd4b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:56:34.911826 21061 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75910e054c2d4704b6961aec0c9fcd4b [term 1 LEADER]: Becoming Leader. State: Replica: 75910e054c2d4704b6961aec0c9fcd4b, State: Running, Role: LEADER
I20260929 15:56:34.912345 21061 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75910e054c2d4704b6961aec0c9fcd4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75910e054c2d4704b6961aec0c9fcd4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46667 } }
I20260929 15:56:34.912964 21051 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75910e054c2d4704b6961aec0c9fcd4b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:56:34.914992 21062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75910e054c2d4704b6961aec0c9fcd4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75910e054c2d4704b6961aec0c9fcd4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75910e054c2d4704b6961aec0c9fcd4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46667 } } }
I20260929 15:56:34.915362 21062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75910e054c2d4704b6961aec0c9fcd4b [sys.catalog]: This master's current role is: LEADER
I20260929 15:56:34.915792 21063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75910e054c2d4704b6961aec0c9fcd4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75910e054c2d4704b6961aec0c9fcd4b. Latest consensus state: current_term: 1 leader_uuid: "75910e054c2d4704b6961aec0c9fcd4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75910e054c2d4704b6961aec0c9fcd4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46667 } } }
I20260929 15:56:34.916127 21063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75910e054c2d4704b6961aec0c9fcd4b [sys.catalog]: This master's current role is: LEADER
I20260929 15:56:34.918062 21068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:56:34.923246 21068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:56:34.932142 21068 catalog_manager.cc:1347] Generated new cluster ID: f437ee6625a74b6291fe9a548c59cfc5
I20260929 15:56:34.932252 21068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:56:34.950397 21068 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:56:34.950821 21068 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:56:34.958523 21068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75910e054c2d4704b6961aec0c9fcd4b: Generated new TSK 0
I20260929 15:56:34.959268 21068 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:56:34.973011 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 20984
W20260929 15:56:34.974050 21046 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47223
W20260929 15:56:34.974874 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 20984: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:56:34.975024 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 20984
2026-09-29T15:56:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1892 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:56:35 dist-test-slave-r0ck krb5kdc[21082](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:56:35 dist-test-slave-r0ck krb5kdc[21082](info): set up 2 sockets
Sep 29 15:56:35 dist-test-slave-r0ck krb5kdc[21082](info): commencing operation
krb5kdc: starting...
W20260929 15:56:36.089715 388 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:56:36 dist-test-slave-r0ck krb5kdc[21082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:56:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:56:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 15:56:36.337630 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38515
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:56:36.666510 21095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:56:36.666832 21095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:56:36.666991 21095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:56:36.674173 21095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:56:36.674321 21095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:56:36.674427 21095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:56:36.674515 21095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:56:36.674582 21095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:56:36.684147 21095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:56:36.685410 21095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:56:36.686707 21095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:56:36.698421 21100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:36.698396 21103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:36.698482 21101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:56:36.699227 21095 server_base.cc:1034] running on GCE node
I20260929 15:56:36.700060 21095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:56:36.701443 21095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:56:36.702669 21095 hybrid_clock.cc:648] HybridClock initialized: now 1790697396702624 us; error 58 us; skew 500 ppm
I20260929 15:56:36.703015 21095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:56:36 dist-test-slave-r0ck krb5kdc[21082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:56:36.706116 21095 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 15:56:36.707470 21095 webserver.cc:460] Webserver started at http://127.0.0.1:35559/ using document root <none> and password file <none>
I20260929 15:56:36.708091 21095 fs_manager.cc:359] Metadata directory not provided
I20260929 15:56:36.708204 21095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:56:36.708441 21095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:56:36.711386 21095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "29be5afce76347d29432389463394b3c"
format_stamp: "Formatted at 2026-09-29 15:56:36 on dist-test-slave-r0ck"
server_key: "82d9ded0d35e3dc8894cd9c0fdfe1bb1"
server_key_iv: "b23c6750655c880ad29854ffdd203cd5"
server_key_version: "encryptionkey@0"
I20260929 15:56:36.712150 21095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "29be5afce76347d29432389463394b3c"
format_stamp: "Formatted at 2026-09-29 15:56:36 on dist-test-slave-r0ck"
server_key: "82d9ded0d35e3dc8894cd9c0fdfe1bb1"
server_key_iv: "b23c6750655c880ad29854ffdd203cd5"
server_key_version: "encryptionkey@0"
I20260929 15:56:36.717145 21095 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 15:56:36.720497 21110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:36.721927 21095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:56:36.722066 21095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "29be5afce76347d29432389463394b3c"
format_stamp: "Formatted at 2026-09-29 15:56:36 on dist-test-slave-r0ck"
server_key: "82d9ded0d35e3dc8894cd9c0fdfe1bb1"
server_key_iv: "b23c6750655c880ad29854ffdd203cd5"
server_key_version: "encryptionkey@0"
I20260929 15:56:36.722230 21095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:56:36.739183 21095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:56:36.743070 21095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:56:36.743299 21095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:56:36.761386 21095 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38515
I20260929 15:56:36.761472 21161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38515 every 8 connection(s)
I20260929 15:56:36.762714 21095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:56:36.764907 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 21095
I20260929 15:56:36.765161 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:56:36.765872 388 external_mini_cluster.cc:1428] Setting key a8f3f4faf97417e2a366f3ead7d4319b
I20260929 15:56:36.766770 21162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 15:56:36 dist-test-slave-r0ck krb5kdc[21082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 15:56:36.780133 21162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29be5afce76347d29432389463394b3c: Bootstrap starting.
I20260929 15:56:36.784041 21162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29be5afce76347d29432389463394b3c: Neither blocks nor log segments found. Creating new log.
I20260929 15:56:36.785249 21162 log.cc:824] T 00000000000000000000000000000000 P 29be5afce76347d29432389463394b3c: Log is configured to *not* fsync() on all Append() calls
I20260929 15:56:36.788371 21162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29be5afce76347d29432389463394b3c: No bootstrap required, opened a new log
I20260929 15:56:36.788626 21165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:36.767752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43199 (local address 127.0.0.1:38515)
0929 15:56:36.768526 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:36.768543 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:36.768590 (+ 47us) server_negotiation.cc:407] Connection header received
0929 15:56:36.770332 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:36.770358 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:36.770866 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:36.771261 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:36.772518 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:36.774530 (+ 2012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:36.775791 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:36.776775 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:36.776953 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:36.780725 (+ 3772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:36.780734 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:36.780750 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:56:36.780789 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:36.784620 (+ 3831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:36.785226 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:36.785240 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:36.785249 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:36.785459 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:36.785849 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:36.785858 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:36.785863 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:36.786376 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:36.786678 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:36.787018 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 15:56:36.787405 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":338,"threads_started":1}
I20260929 15:56:36.792333 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:56:36.796710 21162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29be5afce76347d29432389463394b3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29be5afce76347d29432389463394b3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38515 } }
I20260929 15:56:36.797164 21162 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29be5afce76347d29432389463394b3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:56:36.797294 21162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29be5afce76347d29432389463394b3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29be5afce76347d29432389463394b3c, State: Initialized, Role: FOLLOWER
I20260929 15:56:36.797959 21162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29be5afce76347d29432389463394b3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29be5afce76347d29432389463394b3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38515 } }
I20260929 15:56:36.798256 21162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29be5afce76347d29432389463394b3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:56:36.798398 21162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29be5afce76347d29432389463394b3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:56:36.798589 21162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29be5afce76347d29432389463394b3c [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:56:36.801540 21162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29be5afce76347d29432389463394b3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29be5afce76347d29432389463394b3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38515 } }
I20260929 15:56:36.802110 21162 leader_election.cc:302] T 00000000000000000000000000000000 P 29be5afce76347d29432389463394b3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29be5afce76347d29432389463394b3c; no voters:
I20260929 15:56:36.802753 21162 leader_election.cc:288] T 00000000000000000000000000000000 P 29be5afce76347d29432389463394b3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:56:36.802870 21172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29be5afce76347d29432389463394b3c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:56:36.804148 21172 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29be5afce76347d29432389463394b3c [term 1 LEADER]: Becoming Leader. State: Replica: 29be5afce76347d29432389463394b3c, State: Running, Role: LEADER
I20260929 15:56:36.804764 21172 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29be5afce76347d29432389463394b3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29be5afce76347d29432389463394b3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38515 } }
I20260929 15:56:36.805073 21162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29be5afce76347d29432389463394b3c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:56:36.808882 21174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29be5afce76347d29432389463394b3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29be5afce76347d29432389463394b3c. Latest consensus state: current_term: 1 leader_uuid: "29be5afce76347d29432389463394b3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29be5afce76347d29432389463394b3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38515 } } }
I20260929 15:56:36.809329 21174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29be5afce76347d29432389463394b3c [sys.catalog]: This master's current role is: LEADER
I20260929 15:56:36.809520 21173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29be5afce76347d29432389463394b3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29be5afce76347d29432389463394b3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29be5afce76347d29432389463394b3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38515 } } }
I20260929 15:56:36.809818 21173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29be5afce76347d29432389463394b3c [sys.catalog]: This master's current role is: LEADER
I20260929 15:56:36.812295 21165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:36.796547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60545 (local address 127.0.0.1:38515)
0929 15:56:36.796749 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:36.796757 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:36.797081 (+ 324us) server_negotiation.cc:407] Connection header received
0929 15:56:36.797264 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:36.797269 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:36.797394 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:36.797506 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:36.799399 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:36.800162 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:36.802027 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:36.802546 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:36.802692 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:36.805491 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:36.805497 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:36.805502 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:56:36.805540 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:36.809326 (+ 3786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:36.810216 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:36.810223 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:36.810227 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:36.810472 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:36.810862 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:36.810868 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:36.810871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:36.811138 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:36.811439 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:36.811720 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 15:56:36.811926 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 15:56:36.816087 21180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:56:36.821105 21180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:56:36.830156 21180 catalog_manager.cc:1347] Generated new cluster ID: 085f00dc5b554d349ecb28f4f270bfa6
I20260929 15:56:36.830253 21180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:56:36.860113 21180 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:56:36.862015 21180 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:56:36.871124 21180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29be5afce76347d29432389463394b3c: Generated new TSK 0
I20260929 15:56:36.871958 21180 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:56:36.883332 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 21095
I20260929 15:56:36.884663 21157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:56:36.974915 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 21095
2026-09-29T15:56:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1999 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:56:37 dist-test-slave-r0ck krb5kdc[21194](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:56:37 dist-test-slave-r0ck krb5kdc[21194](info): set up 2 sockets
Sep 29 15:56:37 dist-test-slave-r0ck krb5kdc[21194](info): commencing operation
krb5kdc: starting...
W20260929 15:56:38.073752 388 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:56:38 dist-test-slave-r0ck krb5kdc[21194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:56:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:56:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 15:56:38.324589 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37061
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:56:38.644569 21207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:56:38.644825 21207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:56:38.644914 21207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:56:38.651953 21207 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:56:38.652055 21207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:56:38.652109 21207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:56:38.652170 21207 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:56:38.652217 21207 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:56:38.661717 21207 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:56:38.662861 21207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:56:38.664537 21207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:56:38.675669 21213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:38.676057 21212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:38.676085 21215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:56:38.676433 21207 server_base.cc:1034] running on GCE node
I20260929 15:56:38.677121 21207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:56:38.678413 21207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:56:38.679644 21207 hybrid_clock.cc:648] HybridClock initialized: now 1790697398679611 us; error 40 us; skew 500 ppm
I20260929 15:56:38.679982 21207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:56:38 dist-test-slave-r0ck krb5kdc[21194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:56:38.682893 21207 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 15:56:38.684228 21207 webserver.cc:460] Webserver started at http://127.0.0.1:39229/ using document root <none> and password file <none>
I20260929 15:56:38.684855 21207 fs_manager.cc:359] Metadata directory not provided
I20260929 15:56:38.684996 21207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:56:38.685297 21207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:56:38.688176 21207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "118fdea537f24d96b2a31212e05d0db7"
format_stamp: "Formatted at 2026-09-29 15:56:38 on dist-test-slave-r0ck"
server_key: "b45af53bf3f87f9b90c46fbf7c2eb849"
server_key_iv: "50816c222211f0fc98cc20cd7c3bfdf8"
server_key_version: "encryptionkey@0"
I20260929 15:56:38.688910 21207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "118fdea537f24d96b2a31212e05d0db7"
format_stamp: "Formatted at 2026-09-29 15:56:38 on dist-test-slave-r0ck"
server_key: "b45af53bf3f87f9b90c46fbf7c2eb849"
server_key_iv: "50816c222211f0fc98cc20cd7c3bfdf8"
server_key_version: "encryptionkey@0"
I20260929 15:56:38.693789 21207 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 15:56:38.697078 21222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:38.698338 21207 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:56:38.698513 21207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "118fdea537f24d96b2a31212e05d0db7"
format_stamp: "Formatted at 2026-09-29 15:56:38 on dist-test-slave-r0ck"
server_key: "b45af53bf3f87f9b90c46fbf7c2eb849"
server_key_iv: "50816c222211f0fc98cc20cd7c3bfdf8"
server_key_version: "encryptionkey@0"
I20260929 15:56:38.698705 21207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:56:38.718261 21207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:56:38.722067 21207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:56:38.722357 21207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:56:38.740569 21207 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37061
I20260929 15:56:38.740602 21273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37061 every 8 connection(s)
I20260929 15:56:38.741858 21207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:56:38.745412 21274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:56:38.747068 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 21207
I20260929 15:56:38.747263 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:56:38.747840 388 external_mini_cluster.cc:1428] Setting key 9e70df11d9d255b1baee459556049263
I20260929 15:56:38.758814 21274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 118fdea537f24d96b2a31212e05d0db7: Bootstrap starting.
I20260929 15:56:38.760402 21277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:38.749604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60453 (local address 127.0.0.1:37061)
0929 15:56:38.750373 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:38.750386 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:38.750424 (+ 38us) server_negotiation.cc:407] Connection header received
0929 15:56:38.751976 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:38.751995 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:38.752364 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:38.752650 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:38.753166 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:38.754431 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:38.756332 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:38.756982 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:38.757116 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:38.758276 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:38.758283 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:38.758295 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:38.758789 (+ 494us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:38.758862 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:38.758914 (+ 52us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:38.759050 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:38.759287 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 15:56:38.759531 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":365,"threads_started":1}
I20260929 15:56:38.761888 21274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 118fdea537f24d96b2a31212e05d0db7: Neither blocks nor log segments found. Creating new log.
I20260929 15:56:38.762991 21274 log.cc:824] T 00000000000000000000000000000000 P 118fdea537f24d96b2a31212e05d0db7: Log is configured to *not* fsync() on all Append() calls
I20260929 15:56:38.763826 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:56:38.765822 21274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 118fdea537f24d96b2a31212e05d0db7: No bootstrap required, opened a new log
I20260929 15:56:38.774188 21277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:38.768200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51621 (local address 127.0.0.1:37061)
0929 15:56:38.768369 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:38.768378 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:38.768753 (+ 375us) server_negotiation.cc:407] Connection header received
0929 15:56:38.768945 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:38.768952 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:38.769016 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:38.769131 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:38.769656 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:38.770373 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:38.771811 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:38.772279 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:38.772430 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:38.773491 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:38.773495 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:38.773498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:38.773515 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:38.773548 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:38.773563 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:38.773660 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:38.773811 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 15:56:38.773897 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 15:56:38.773904 21274 raft_consensus.cc:348] T 00000000000000000000000000000000 P 118fdea537f24d96b2a31212e05d0db7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118fdea537f24d96b2a31212e05d0db7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37061 } }
I20260929 15:56:38.774397 21274 raft_consensus.cc:374] T 00000000000000000000000000000000 P 118fdea537f24d96b2a31212e05d0db7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:56:38.774533 21274 raft_consensus.cc:729] T 00000000000000000000000000000000 P 118fdea537f24d96b2a31212e05d0db7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 118fdea537f24d96b2a31212e05d0db7, State: Initialized, Role: FOLLOWER
I20260929 15:56:38.775218 21274 consensus_queue.cc:260] T 00000000000000000000000000000000 P 118fdea537f24d96b2a31212e05d0db7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118fdea537f24d96b2a31212e05d0db7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37061 } }
I20260929 15:56:38.775511 21274 raft_consensus.cc:388] T 00000000000000000000000000000000 P 118fdea537f24d96b2a31212e05d0db7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:56:38.775667 21274 raft_consensus.cc:482] T 00000000000000000000000000000000 P 118fdea537f24d96b2a31212e05d0db7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:56:38.775846 21274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 118fdea537f24d96b2a31212e05d0db7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:56:38.778031 21274 raft_consensus.cc:504] T 00000000000000000000000000000000 P 118fdea537f24d96b2a31212e05d0db7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118fdea537f24d96b2a31212e05d0db7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37061 } }
I20260929 15:56:38.778502 21274 leader_election.cc:302] T 00000000000000000000000000000000 P 118fdea537f24d96b2a31212e05d0db7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 118fdea537f24d96b2a31212e05d0db7; no voters:
I20260929 15:56:38.779021 21274 leader_election.cc:288] T 00000000000000000000000000000000 P 118fdea537f24d96b2a31212e05d0db7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:56:38.779168 21284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 118fdea537f24d96b2a31212e05d0db7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:56:38.780627 21284 raft_consensus.cc:686] T 00000000000000000000000000000000 P 118fdea537f24d96b2a31212e05d0db7 [term 1 LEADER]: Becoming Leader. State: Replica: 118fdea537f24d96b2a31212e05d0db7, State: Running, Role: LEADER
I20260929 15:56:38.781148 21284 consensus_queue.cc:237] T 00000000000000000000000000000000 P 118fdea537f24d96b2a31212e05d0db7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118fdea537f24d96b2a31212e05d0db7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37061 } }
I20260929 15:56:38.781391 21274 sys_catalog.cc:564] T 00000000000000000000000000000000 P 118fdea537f24d96b2a31212e05d0db7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:56:38.783743 21286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 118fdea537f24d96b2a31212e05d0db7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 118fdea537f24d96b2a31212e05d0db7. Latest consensus state: current_term: 1 leader_uuid: "118fdea537f24d96b2a31212e05d0db7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118fdea537f24d96b2a31212e05d0db7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37061 } } }
I20260929 15:56:38.783838 21285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 118fdea537f24d96b2a31212e05d0db7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "118fdea537f24d96b2a31212e05d0db7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118fdea537f24d96b2a31212e05d0db7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37061 } } }
I20260929 15:56:38.784125 21286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 118fdea537f24d96b2a31212e05d0db7 [sys.catalog]: This master's current role is: LEADER
I20260929 15:56:38.784207 21285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 118fdea537f24d96b2a31212e05d0db7 [sys.catalog]: This master's current role is: LEADER
I20260929 15:56:38.786481 21290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:56:38.791941 21290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:56:38.800674 21290 catalog_manager.cc:1347] Generated new cluster ID: 172ad4e27ccf41bca23548ecddcefbfc
I20260929 15:56:38.800868 21290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:56:38.823799 21290 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:56:38.825052 21290 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:56:38.832230 21290 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 118fdea537f24d96b2a31212e05d0db7: Generated new TSK 0
I20260929 15:56:38.832991 21290 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:56:38.843533 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 21207
W20260929 15:56:38.844489 21269 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60453
W20260929 15:56:38.845130 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 21207: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:56:38.845248 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 21207
2026-09-29T15:56:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1867 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:56:38 dist-test-slave-r0ck krb5kdc[21305](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:56:38 dist-test-slave-r0ck krb5kdc[21305](info): set up 2 sockets
Sep 29 15:56:38 dist-test-slave-r0ck krb5kdc[21305](info): commencing operation
krb5kdc: starting...
W20260929 15:56:39.933728 388 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:56:40 dist-test-slave-r0ck krb5kdc[21305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:56:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:56:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 15:56:40.181653 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44459
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:56:40.498956 21318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:56:40.499218 21318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:56:40.499382 21318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:56:40.506502 21318 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:56:40.506604 21318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:56:40.506744 21318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:56:40.506843 21318 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:56:40.506906 21318 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:56:40.518028 21318 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:56:40.519258 21318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:56:40.520486 21318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:56:40.537215 21324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:40.537207 21323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:40.537521 21326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:56:40.538090 21318 server_base.cc:1034] running on GCE node
I20260929 15:56:40.538951 21318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:56:40.540954 21318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:56:40.542167 21318 hybrid_clock.cc:648] HybridClock initialized: now 1790697400542086 us; error 43 us; skew 500 ppm
I20260929 15:56:40.542645 21318 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:56:40 dist-test-slave-r0ck krb5kdc[21305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:56:40.546922 21318 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 15:56:40.548856 21318 webserver.cc:460] Webserver started at http://127.0.0.1:36385/ using document root <none> and password file <none>
I20260929 15:56:40.549774 21318 fs_manager.cc:359] Metadata directory not provided
I20260929 15:56:40.549919 21318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:56:40.550276 21318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:56:40.553928 21318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "ef6e463e5dfa460d92c72ed268a372e3"
format_stamp: "Formatted at 2026-09-29 15:56:40 on dist-test-slave-r0ck"
server_key: "d72687171388512fdc1a825fd1a54fe6"
server_key_iv: "749b4ba8944a9a965f66454d25b1c8f3"
server_key_version: "encryptionkey@0"
I20260929 15:56:40.554692 21318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "ef6e463e5dfa460d92c72ed268a372e3"
format_stamp: "Formatted at 2026-09-29 15:56:40 on dist-test-slave-r0ck"
server_key: "d72687171388512fdc1a825fd1a54fe6"
server_key_iv: "749b4ba8944a9a965f66454d25b1c8f3"
server_key_version: "encryptionkey@0"
I20260929 15:56:40.562260 21318 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260929 15:56:40.566601 21333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:40.568303 21318 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 15:56:40.568480 21318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "ef6e463e5dfa460d92c72ed268a372e3"
format_stamp: "Formatted at 2026-09-29 15:56:40 on dist-test-slave-r0ck"
server_key: "d72687171388512fdc1a825fd1a54fe6"
server_key_iv: "749b4ba8944a9a965f66454d25b1c8f3"
server_key_version: "encryptionkey@0"
I20260929 15:56:40.568696 21318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:56:40.575002 21318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:56:40.579276 21318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:56:40.579561 21318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:56:40.597944 21318 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44459
I20260929 15:56:40.597991 21384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44459 every 8 connection(s)
I20260929 15:56:40.599272 21318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:56:40.602833 21385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:56:40.603860 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 21318
I20260929 15:56:40.604055 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:56:40.604501 388 external_mini_cluster.cc:1428] Setting key fd0cad3d39a27b05f630a875fb8f65cc
I20260929 15:56:40.611593 21388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:40.606170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53437 (local address 127.0.0.1:44459)
0929 15:56:40.606930 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:40.606941 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:40.606980 (+ 39us) server_negotiation.cc:407] Connection header received
0929 15:56:40.608532 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:40.608554 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:40.608854 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:40.609165 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:40.610209 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:40.610215 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:40.610226 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:40.610239 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:40.610293 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:40.610314 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:40.610469 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:40.610583 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 15:56:40.610775 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":370,"threads_started":1}
I20260929 15:56:40.615154 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:56:40.615687 21385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef6e463e5dfa460d92c72ed268a372e3: Bootstrap starting.
I20260929 15:56:40.618724 21385 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef6e463e5dfa460d92c72ed268a372e3: Neither blocks nor log segments found. Creating new log.
I20260929 15:56:40.619881 21385 log.cc:824] T 00000000000000000000000000000000 P ef6e463e5dfa460d92c72ed268a372e3: Log is configured to *not* fsync() on all Append() calls
I20260929 15:56:40.622007 21388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:40.619651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49477 (local address 127.0.0.1:44459)
0929 15:56:40.619839 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:40.619844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:40.620126 (+ 282us) server_negotiation.cc:407] Connection header received
0929 15:56:40.620292 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:40.620295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:40.620365 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:40.620487 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:40.621374 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:40.621379 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:40.621384 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:40.621404 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:40.621448 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:40.621467 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:40.621584 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:40.621659 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 15:56:40.621719 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 15:56:40.623001 21385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef6e463e5dfa460d92c72ed268a372e3: No bootstrap required, opened a new log
I20260929 15:56:40.630616 21385 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef6e463e5dfa460d92c72ed268a372e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef6e463e5dfa460d92c72ed268a372e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44459 } }
I20260929 15:56:40.631033 21385 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef6e463e5dfa460d92c72ed268a372e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:56:40.631142 21385 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef6e463e5dfa460d92c72ed268a372e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef6e463e5dfa460d92c72ed268a372e3, State: Initialized, Role: FOLLOWER
I20260929 15:56:40.631829 21385 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef6e463e5dfa460d92c72ed268a372e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef6e463e5dfa460d92c72ed268a372e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44459 } }
I20260929 15:56:40.632037 21385 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef6e463e5dfa460d92c72ed268a372e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:56:40.632134 21385 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef6e463e5dfa460d92c72ed268a372e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:56:40.632259 21385 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef6e463e5dfa460d92c72ed268a372e3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:56:40.634455 21385 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef6e463e5dfa460d92c72ed268a372e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef6e463e5dfa460d92c72ed268a372e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44459 } }
I20260929 15:56:40.634872 21385 leader_election.cc:302] T 00000000000000000000000000000000 P ef6e463e5dfa460d92c72ed268a372e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef6e463e5dfa460d92c72ed268a372e3; no voters:
I20260929 15:56:40.635551 21385 leader_election.cc:288] T 00000000000000000000000000000000 P ef6e463e5dfa460d92c72ed268a372e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:56:40.635659 21395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef6e463e5dfa460d92c72ed268a372e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:56:40.636322 21395 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef6e463e5dfa460d92c72ed268a372e3 [term 1 LEADER]: Becoming Leader. State: Replica: ef6e463e5dfa460d92c72ed268a372e3, State: Running, Role: LEADER
I20260929 15:56:40.636806 21395 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef6e463e5dfa460d92c72ed268a372e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef6e463e5dfa460d92c72ed268a372e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44459 } }
I20260929 15:56:40.637622 21385 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef6e463e5dfa460d92c72ed268a372e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:56:40.639654 21396 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef6e463e5dfa460d92c72ed268a372e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef6e463e5dfa460d92c72ed268a372e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef6e463e5dfa460d92c72ed268a372e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44459 } } }
I20260929 15:56:40.639573 21397 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef6e463e5dfa460d92c72ed268a372e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef6e463e5dfa460d92c72ed268a372e3. Latest consensus state: current_term: 1 leader_uuid: "ef6e463e5dfa460d92c72ed268a372e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef6e463e5dfa460d92c72ed268a372e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44459 } } }
I20260929 15:56:40.640133 21396 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef6e463e5dfa460d92c72ed268a372e3 [sys.catalog]: This master's current role is: LEADER
I20260929 15:56:40.640175 21397 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef6e463e5dfa460d92c72ed268a372e3 [sys.catalog]: This master's current role is: LEADER
I20260929 15:56:40.641944 21400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:56:40.648913 21400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:56:40.657713 21400 catalog_manager.cc:1347] Generated new cluster ID: 0f16b511c394423886c2c14e9ef172a3
I20260929 15:56:40.657846 21400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:56:40.670585 21400 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:56:40.670969 21400 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:56:40.679435 21400 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef6e463e5dfa460d92c72ed268a372e3: Generated new TSK 0
I20260929 15:56:40.680145 21400 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:56:40.685503 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 21318
W20260929 15:56:40.686705 21380 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53437
W20260929 15:56:40.687518 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 21318: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:56:40.687666 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 21318
2026-09-29T15:56:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1844 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:56:40 dist-test-slave-r0ck krb5kdc[21416](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:56:40 dist-test-slave-r0ck krb5kdc[21416](info): set up 2 sockets
Sep 29 15:56:40 dist-test-slave-r0ck krb5kdc[21416](info): commencing operation
krb5kdc: starting...
W20260929 15:56:41.777710 388 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:56:41 dist-test-slave-r0ck krb5kdc[21416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:56:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:56:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 15:56:42.044181 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35011
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35011
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:56:42.364131 21429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:56:42.364415 21429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:56:42.364557 21429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:56:42.371692 21429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:56:42.371796 21429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:56:42.371904 21429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:56:42.372014 21429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:56:42.372092 21429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:56:42.382020 21429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:56:42.383255 21429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:56:42.384516 21429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:56:42.395962 21434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:42.396034 21435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:42.396054 21437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:56:42.397018 21429 server_base.cc:1034] running on GCE node
I20260929 15:56:42.397902 21429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:56:42.399281 21429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:56:42.400521 21429 hybrid_clock.cc:648] HybridClock initialized: now 1790697402400480 us; error 35 us; skew 500 ppm
I20260929 15:56:42.400890 21429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:56:42 dist-test-slave-r0ck krb5kdc[21416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:56:42.404038 21429 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 15:56:42.405566 21429 webserver.cc:460] Webserver started at http://127.0.0.1:41469/ using document root <none> and password file <none>
I20260929 15:56:42.406232 21429 fs_manager.cc:359] Metadata directory not provided
I20260929 15:56:42.406384 21429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:56:42.406687 21429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:56:42.409605 21429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "8802e29013574ca7b96a965c1f24b438"
format_stamp: "Formatted at 2026-09-29 15:56:42 on dist-test-slave-r0ck"
server_key: "0d1b6c6f5a557504f3927e1d8cd2f4c4"
server_key_iv: "332c591a3a442f513925fcdfccaed597"
server_key_version: "encryptionkey@0"
I20260929 15:56:42.410319 21429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "8802e29013574ca7b96a965c1f24b438"
format_stamp: "Formatted at 2026-09-29 15:56:42 on dist-test-slave-r0ck"
server_key: "0d1b6c6f5a557504f3927e1d8cd2f4c4"
server_key_iv: "332c591a3a442f513925fcdfccaed597"
server_key_version: "encryptionkey@0"
I20260929 15:56:42.415167 21429 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 15:56:42.418494 21444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:42.419736 21429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:56:42.419866 21429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "8802e29013574ca7b96a965c1f24b438"
format_stamp: "Formatted at 2026-09-29 15:56:42 on dist-test-slave-r0ck"
server_key: "0d1b6c6f5a557504f3927e1d8cd2f4c4"
server_key_iv: "332c591a3a442f513925fcdfccaed597"
server_key_version: "encryptionkey@0"
I20260929 15:56:42.420017 21429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:56:42.438740 21429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:56:42.442972 21429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:56:42.443214 21429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:56:42.461963 21429 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35011
I20260929 15:56:42.462002 21495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35011 every 8 connection(s)
I20260929 15:56:42.463347 21429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:56:42.467060 21496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:56:42.467649 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 21429
I20260929 15:56:42.467849 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:56:42.468300 388 external_mini_cluster.cc:1428] Setting key 27314645707f5f2ed9b85437a6f8deee
I20260929 15:56:42.480719 21496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8802e29013574ca7b96a965c1f24b438: Bootstrap starting.
Sep 29 15:56:42 dist-test-slave-r0ck krb5kdc[21416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 15:56:42.484367 21496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8802e29013574ca7b96a965c1f24b438: Neither blocks nor log segments found. Creating new log.
I20260929 15:56:42.485671 21496 log.cc:824] T 00000000000000000000000000000000 P 8802e29013574ca7b96a965c1f24b438: Log is configured to *not* fsync() on all Append() calls
I20260929 15:56:42.488766 21496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8802e29013574ca7b96a965c1f24b438: No bootstrap required, opened a new log
I20260929 15:56:42.492179 21499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:42.469993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.189:33685 (local address 127.0.0.1:35011)
0929 15:56:42.470755 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:42.470769 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:42.470809 (+ 40us) server_negotiation.cc:407] Connection header received
0929 15:56:42.473026 (+ 2217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:42.473048 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:42.473526 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:42.473835 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:42.475503 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:42.476774 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:42.478599 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:42.479258 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:42.479406 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:42.484233 (+ 4827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:42.484241 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:42.484254 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:56:42.484294 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:42.487960 (+ 3666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:42.488799 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:42.488805 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:42.488811 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:42.489016 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:42.489528 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:42.489533 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:42.489537 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:42.490035 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:42.490385 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:42.490897 (+ 512us) server_negotiation.cc:299] Negotiation successful
0929 15:56:42.491208 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":375,"threads_started":1}
I20260929 15:56:42.495990 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:56:42.496966 21496 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8802e29013574ca7b96a965c1f24b438 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8802e29013574ca7b96a965c1f24b438" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35011 } }
I20260929 15:56:42.497376 21496 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8802e29013574ca7b96a965c1f24b438 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:56:42.497490 21496 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8802e29013574ca7b96a965c1f24b438 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8802e29013574ca7b96a965c1f24b438, State: Initialized, Role: FOLLOWER
I20260929 15:56:42.498018 21496 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8802e29013574ca7b96a965c1f24b438 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8802e29013574ca7b96a965c1f24b438" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35011 } }
I20260929 15:56:42.498266 21496 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8802e29013574ca7b96a965c1f24b438 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:56:42.498409 21496 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8802e29013574ca7b96a965c1f24b438 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:56:42.498597 21496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8802e29013574ca7b96a965c1f24b438 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:56:42.501230 21496 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8802e29013574ca7b96a965c1f24b438 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8802e29013574ca7b96a965c1f24b438" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35011 } }
I20260929 15:56:42.501797 21496 leader_election.cc:302] T 00000000000000000000000000000000 P 8802e29013574ca7b96a965c1f24b438 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8802e29013574ca7b96a965c1f24b438; no voters:
I20260929 15:56:42.502391 21496 leader_election.cc:288] T 00000000000000000000000000000000 P 8802e29013574ca7b96a965c1f24b438 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:56:42.502561 21506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8802e29013574ca7b96a965c1f24b438 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:56:42.503194 21506 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8802e29013574ca7b96a965c1f24b438 [term 1 LEADER]: Becoming Leader. State: Replica: 8802e29013574ca7b96a965c1f24b438, State: Running, Role: LEADER
I20260929 15:56:42.503803 21506 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8802e29013574ca7b96a965c1f24b438 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8802e29013574ca7b96a965c1f24b438" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35011 } }
I20260929 15:56:42.506569 21508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8802e29013574ca7b96a965c1f24b438 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8802e29013574ca7b96a965c1f24b438" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8802e29013574ca7b96a965c1f24b438" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35011 } } }
I20260929 15:56:42.506978 21508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8802e29013574ca7b96a965c1f24b438 [sys.catalog]: This master's current role is: LEADER
I20260929 15:56:42.509261 21507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8802e29013574ca7b96a965c1f24b438 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8802e29013574ca7b96a965c1f24b438. Latest consensus state: current_term: 1 leader_uuid: "8802e29013574ca7b96a965c1f24b438" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8802e29013574ca7b96a965c1f24b438" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35011 } } }
I20260929 15:56:42.509629 21507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8802e29013574ca7b96a965c1f24b438 [sys.catalog]: This master's current role is: LEADER
I20260929 15:56:42.512405 21496 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8802e29013574ca7b96a965c1f24b438 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:56:42.515437 21510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:56:42.520471 21499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:42.500435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.189:43271 (local address 127.0.0.1:35011)
0929 15:56:42.500625 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:42.500631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:42.501100 (+ 469us) server_negotiation.cc:407] Connection header received
0929 15:56:42.501238 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:42.501245 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:42.501332 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:42.501471 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:42.503321 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:42.504474 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:42.506555 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:42.507337 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:42.507476 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:42.511866 (+ 4390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:42.511876 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:42.511884 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:56:42.511941 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:42.516538 (+ 4597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:42.517662 (+ 1124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:42.517668 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:42.517671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:42.517818 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:42.518412 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:42.518418 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:42.518422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:42.518865 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:42.519132 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:42.519901 (+ 769us) server_negotiation.cc:299] Negotiation successful
0929 15:56:42.520110 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 15:56:42.523921 21510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:56:42.538707 21510 catalog_manager.cc:1347] Generated new cluster ID: 13f5762b077748e888403a5c01431216
I20260929 15:56:42.538859 21510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:56:42.559268 21510 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:56:42.561120 21510 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:56:42.572089 21510 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8802e29013574ca7b96a965c1f24b438: Generated new TSK 0
I20260929 15:56:42.573076 21510 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:56:42.578030 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 21429
I20260929 15:56:42.580005 21491 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:56:42.675334 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 21429
W20260929 15:56:42.694398 21414 connection.cc:535] client connection to 127.0.0.1:35011 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35011): Connection reset by peer (error 104)
2026-09-29T15:56:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1990 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:56:42 dist-test-slave-r0ck krb5kdc[21529](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:56:42 dist-test-slave-r0ck krb5kdc[21529](info): set up 2 sockets
Sep 29 15:56:42 dist-test-slave-r0ck krb5kdc[21529](info): commencing operation
krb5kdc: starting...
W20260929 15:56:43.789665 388 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:56:43 dist-test-slave-r0ck krb5kdc[21529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:56:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:56:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 15:56:44.058632 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35985
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:56:44.378810 21542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:56:44.379076 21542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:56:44.379276 21542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:56:44.386309 21542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:56:44.386406 21542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:56:44.386525 21542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:56:44.386622 21542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:56:44.386704 21542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:56:44.396174 21542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:56:44.397459 21542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:56:44.398691 21542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:56:44.411350 21550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:44.411456 21548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:44.411434 21547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:56:44.412218 21542 server_base.cc:1034] running on GCE node
I20260929 15:56:44.412940 21542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:56:44.414371 21542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:56:44.415565 21542 hybrid_clock.cc:648] HybridClock initialized: now 1790697404415507 us; error 50 us; skew 500 ppm
I20260929 15:56:44.415915 21542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:56:44 dist-test-slave-r0ck krb5kdc[21529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:56:44.419014 21542 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 15:56:44.420463 21542 webserver.cc:460] Webserver started at http://127.0.0.1:40291/ using document root <none> and password file <none>
I20260929 15:56:44.421185 21542 fs_manager.cc:359] Metadata directory not provided
I20260929 15:56:44.421324 21542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:56:44.421666 21542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:56:44.424887 21542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "2a35297af3424b9bb64f02684e52f64f"
format_stamp: "Formatted at 2026-09-29 15:56:44 on dist-test-slave-r0ck"
server_key: "19677f0ac7725ac13e654b7d7adaab0f"
server_key_iv: "bd0161bd4d8159a40c451e07e3f8d04d"
server_key_version: "encryptionkey@0"
I20260929 15:56:44.425737 21542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "2a35297af3424b9bb64f02684e52f64f"
format_stamp: "Formatted at 2026-09-29 15:56:44 on dist-test-slave-r0ck"
server_key: "19677f0ac7725ac13e654b7d7adaab0f"
server_key_iv: "bd0161bd4d8159a40c451e07e3f8d04d"
server_key_version: "encryptionkey@0"
I20260929 15:56:44.430809 21542 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260929 15:56:44.434198 21557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:44.435510 21542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 15:56:44.435701 21542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "2a35297af3424b9bb64f02684e52f64f"
format_stamp: "Formatted at 2026-09-29 15:56:44 on dist-test-slave-r0ck"
server_key: "19677f0ac7725ac13e654b7d7adaab0f"
server_key_iv: "bd0161bd4d8159a40c451e07e3f8d04d"
server_key_version: "encryptionkey@0"
I20260929 15:56:44.435899 21542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:56:44.456817 21542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:56:44.461674 21542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:56:44.461977 21542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:56:44.481062 21542 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35985
I20260929 15:56:44.481094 21608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35985 every 8 connection(s)
I20260929 15:56:44.482415 21542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:56:44.486131 21609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:56:44.492344 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 21542
I20260929 15:56:44.492537 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:56:44.493075 388 external_mini_cluster.cc:1428] Setting key 334d5520ed5870eb144f615750f08125
I20260929 15:56:44.497700 21609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a35297af3424b9bb64f02684e52f64f: Bootstrap starting.
I20260929 15:56:44.501084 21609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a35297af3424b9bb64f02684e52f64f: Neither blocks nor log segments found. Creating new log.
I20260929 15:56:44.502125 21609 log.cc:824] T 00000000000000000000000000000000 P 2a35297af3424b9bb64f02684e52f64f: Log is configured to *not* fsync() on all Append() calls
I20260929 15:56:44.505174 21609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a35297af3424b9bb64f02684e52f64f: No bootstrap required, opened a new log
I20260929 15:56:44.506078 21612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:44.495121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.189:43581 (local address 127.0.0.1:35985)
0929 15:56:44.495906 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:44.495918 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:44.495956 (+ 38us) server_negotiation.cc:407] Connection header received
0929 15:56:44.497735 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:44.497756 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:44.498190 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:44.498564 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:44.499096 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:44.500372 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:44.501793 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:44.502531 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:44.502754 (+ 223us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:44.503902 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:44.503909 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:44.503922 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:44.504426 (+ 504us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:44.504493 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:44.504516 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:44.504640 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:44.504904 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 15:56:44.505090 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":334,"threads_started":1}
I20260929 15:56:44.509776 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:56:44.514997 21609 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a35297af3424b9bb64f02684e52f64f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a35297af3424b9bb64f02684e52f64f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35985 } }
I20260929 15:56:44.515503 21609 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a35297af3424b9bb64f02684e52f64f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:56:44.515643 21609 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a35297af3424b9bb64f02684e52f64f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a35297af3424b9bb64f02684e52f64f, State: Initialized, Role: FOLLOWER
I20260929 15:56:44.516433 21609 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a35297af3424b9bb64f02684e52f64f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a35297af3424b9bb64f02684e52f64f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35985 } }
I20260929 15:56:44.516919 21609 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a35297af3424b9bb64f02684e52f64f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:56:44.517186 21609 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a35297af3424b9bb64f02684e52f64f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:56:44.517475 21609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a35297af3424b9bb64f02684e52f64f [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:56:44.520915 21609 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a35297af3424b9bb64f02684e52f64f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a35297af3424b9bb64f02684e52f64f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35985 } }
I20260929 15:56:44.521736 21609 leader_election.cc:302] T 00000000000000000000000000000000 P 2a35297af3424b9bb64f02684e52f64f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a35297af3424b9bb64f02684e52f64f; no voters:
I20260929 15:56:44.522482 21609 leader_election.cc:288] T 00000000000000000000000000000000 P 2a35297af3424b9bb64f02684e52f64f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:56:44.522621 21619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a35297af3424b9bb64f02684e52f64f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:56:44.523207 21612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:44.515261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.189:47505 (local address 127.0.0.1:35985)
0929 15:56:44.515466 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:44.515474 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:44.516027 (+ 553us) server_negotiation.cc:407] Connection header received
0929 15:56:44.516238 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:44.516241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:44.516306 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:44.516388 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:44.517120 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:44.518052 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:44.519958 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:44.520443 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:44.520550 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:44.521968 (+ 1418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:44.521978 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:44.521985 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:44.522009 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:44.522060 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:44.522078 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:44.522178 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:44.522488 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 15:56:44.522553 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 15:56:44.524058 21619 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a35297af3424b9bb64f02684e52f64f [term 1 LEADER]: Becoming Leader. State: Replica: 2a35297af3424b9bb64f02684e52f64f, State: Running, Role: LEADER
I20260929 15:56:44.524775 21619 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a35297af3424b9bb64f02684e52f64f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a35297af3424b9bb64f02684e52f64f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35985 } }
I20260929 15:56:44.525139 21609 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a35297af3424b9bb64f02684e52f64f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:56:44.528285 21620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a35297af3424b9bb64f02684e52f64f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a35297af3424b9bb64f02684e52f64f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a35297af3424b9bb64f02684e52f64f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35985 } } }
I20260929 15:56:44.528777 21620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a35297af3424b9bb64f02684e52f64f [sys.catalog]: This master's current role is: LEADER
I20260929 15:56:44.528251 21621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a35297af3424b9bb64f02684e52f64f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a35297af3424b9bb64f02684e52f64f. Latest consensus state: current_term: 1 leader_uuid: "2a35297af3424b9bb64f02684e52f64f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a35297af3424b9bb64f02684e52f64f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35985 } } }
I20260929 15:56:44.529062 21621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a35297af3424b9bb64f02684e52f64f [sys.catalog]: This master's current role is: LEADER
I20260929 15:56:44.530009 21627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:56:44.536625 21627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:56:44.545755 21627 catalog_manager.cc:1347] Generated new cluster ID: 11da494c2e6f42a0a3d4a24849dbc1b8
I20260929 15:56:44.545881 21627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:56:44.564877 21627 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:56:44.566682 21627 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:56:44.576805 21627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a35297af3424b9bb64f02684e52f64f: Generated new TSK 0
I20260929 15:56:44.577809 21627 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:56:44.585072 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 21542
W20260929 15:56:44.586196 21604 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.189:43581
W20260929 15:56:44.586898 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 21542: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:56:44.587033 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 21542
W20260929 15:56:44.603652 21527 connection.cc:535] client connection to 127.0.0.1:35985 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35985): Connection reset by peer (error 104)
2026-09-29T15:56:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1909 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:56:44 dist-test-slave-r0ck krb5kdc[21640](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:56:44 dist-test-slave-r0ck krb5kdc[21640](info): set up 2 sockets
Sep 29 15:56:44 dist-test-slave-r0ck krb5kdc[21640](info): commencing operation
krb5kdc: starting...
W20260929 15:56:45.705773 388 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:56:45 dist-test-slave-r0ck krb5kdc[21640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:56:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:56:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 15:56:45.973819 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36847
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:56:46.304492 21653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:56:46.304811 21653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:56:46.304972 21653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:56:46.312201 21653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:56:46.312369 21653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:56:46.312471 21653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:56:46.312579 21653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:56:46.312654 21653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:56:46.322757 21653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:56:46.324064 21653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:56:46.325384 21653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:56:46.337828 21661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:46.337798 21659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:46.337906 21658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:56:46.343219 21653 server_base.cc:1034] running on GCE node
I20260929 15:56:46.344139 21653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:56:46.345706 21653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:56:46.346954 21653 hybrid_clock.cc:648] HybridClock initialized: now 1790697406346912 us; error 33 us; skew 500 ppm
I20260929 15:56:46.347332 21653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:56:46 dist-test-slave-r0ck krb5kdc[21640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:56:46.354544 21653 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 15:56:46.356139 21653 webserver.cc:460] Webserver started at http://127.0.0.1:36951/ using document root <none> and password file <none>
I20260929 15:56:46.356863 21653 fs_manager.cc:359] Metadata directory not provided
I20260929 15:56:46.357017 21653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:56:46.357331 21653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:56:46.360426 21653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "9022992314e44544b8a90a44a481aec7"
format_stamp: "Formatted at 2026-09-29 15:56:46 on dist-test-slave-r0ck"
server_key: "7013e601efae98687446921da8e60036"
server_key_iv: "f9d2d0b879a9ae11efe5bed279016268"
server_key_version: "encryptionkey@0"
I20260929 15:56:46.361245 21653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "9022992314e44544b8a90a44a481aec7"
format_stamp: "Formatted at 2026-09-29 15:56:46 on dist-test-slave-r0ck"
server_key: "7013e601efae98687446921da8e60036"
server_key_iv: "f9d2d0b879a9ae11efe5bed279016268"
server_key_version: "encryptionkey@0"
I20260929 15:56:46.368213 21653 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.001s
I20260929 15:56:46.372661 21668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:46.374118 21653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 15:56:46.374323 21653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "9022992314e44544b8a90a44a481aec7"
format_stamp: "Formatted at 2026-09-29 15:56:46 on dist-test-slave-r0ck"
server_key: "7013e601efae98687446921da8e60036"
server_key_iv: "f9d2d0b879a9ae11efe5bed279016268"
server_key_version: "encryptionkey@0"
I20260929 15:56:46.374524 21653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:56:46.380167 21653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:56:46.384251 21653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:56:46.384544 21653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:56:46.402802 21653 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36847
I20260929 15:56:46.402851 21719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36847 every 8 connection(s)
I20260929 15:56:46.404204 21653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:56:46.408778 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 21653
I20260929 15:56:46.409025 21720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:56:46.409060 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:56:46.409631 388 external_mini_cluster.cc:1428] Setting key 5a39cc2bc584b2425e6cb83782cc2a1c
I20260929 15:56:46.417301 21723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:46.411483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.189:50591 (local address 127.0.0.1:36847)
0929 15:56:46.412301 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:46.412312 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:46.412352 (+ 40us) server_negotiation.cc:407] Connection header received
0929 15:56:46.414042 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:46.414062 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:46.414380 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:46.414760 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:46.415791 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:46.415796 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:46.415809 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:46.415822 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:46.415876 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:46.415894 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:46.416133 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:46.416285 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 15:56:46.416451 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":364,"threads_started":1}
I20260929 15:56:46.420766 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:56:46.422302 21720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9022992314e44544b8a90a44a481aec7: Bootstrap starting.
I20260929 15:56:46.425509 21720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9022992314e44544b8a90a44a481aec7: Neither blocks nor log segments found. Creating new log.
I20260929 15:56:46.426806 21720 log.cc:824] T 00000000000000000000000000000000 P 9022992314e44544b8a90a44a481aec7: Log is configured to *not* fsync() on all Append() calls
I20260929 15:56:46.427851 21723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:46.425227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.189:34345 (local address 127.0.0.1:36847)
0929 15:56:46.425456 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:46.425461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:46.425782 (+ 321us) server_negotiation.cc:407] Connection header received
0929 15:56:46.425960 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:46.425963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:46.426012 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:46.426093 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:46.427038 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:46.427050 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:46.427053 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:46.427068 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:46.427106 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:46.427150 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:46.427303 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:46.427438 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 15:56:46.427512 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 15:56:46.429829 21720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9022992314e44544b8a90a44a481aec7: No bootstrap required, opened a new log
I20260929 15:56:46.436889 21720 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9022992314e44544b8a90a44a481aec7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9022992314e44544b8a90a44a481aec7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36847 } }
I20260929 15:56:46.437207 21720 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9022992314e44544b8a90a44a481aec7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:56:46.437290 21720 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9022992314e44544b8a90a44a481aec7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9022992314e44544b8a90a44a481aec7, State: Initialized, Role: FOLLOWER
I20260929 15:56:46.437858 21720 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9022992314e44544b8a90a44a481aec7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9022992314e44544b8a90a44a481aec7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36847 } }
I20260929 15:56:46.438047 21720 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9022992314e44544b8a90a44a481aec7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:56:46.438143 21720 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9022992314e44544b8a90a44a481aec7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:56:46.438265 21720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9022992314e44544b8a90a44a481aec7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:56:46.440451 21720 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9022992314e44544b8a90a44a481aec7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9022992314e44544b8a90a44a481aec7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36847 } }
I20260929 15:56:46.440862 21720 leader_election.cc:302] T 00000000000000000000000000000000 P 9022992314e44544b8a90a44a481aec7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9022992314e44544b8a90a44a481aec7; no voters:
I20260929 15:56:46.441411 21720 leader_election.cc:288] T 00000000000000000000000000000000 P 9022992314e44544b8a90a44a481aec7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:56:46.441566 21730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9022992314e44544b8a90a44a481aec7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:56:46.443303 21720 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9022992314e44544b8a90a44a481aec7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:56:46.445140 21730 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9022992314e44544b8a90a44a481aec7 [term 1 LEADER]: Becoming Leader. State: Replica: 9022992314e44544b8a90a44a481aec7, State: Running, Role: LEADER
I20260929 15:56:46.445714 21730 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9022992314e44544b8a90a44a481aec7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9022992314e44544b8a90a44a481aec7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36847 } }
I20260929 15:56:46.448303 21731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9022992314e44544b8a90a44a481aec7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9022992314e44544b8a90a44a481aec7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9022992314e44544b8a90a44a481aec7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36847 } } }
I20260929 15:56:46.448662 21731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9022992314e44544b8a90a44a481aec7 [sys.catalog]: This master's current role is: LEADER
I20260929 15:56:46.449600 21738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:56:46.449442 21732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9022992314e44544b8a90a44a481aec7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9022992314e44544b8a90a44a481aec7. Latest consensus state: current_term: 1 leader_uuid: "9022992314e44544b8a90a44a481aec7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9022992314e44544b8a90a44a481aec7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36847 } } }
I20260929 15:56:46.449771 21732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9022992314e44544b8a90a44a481aec7 [sys.catalog]: This master's current role is: LEADER
I20260929 15:56:46.455380 21738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:56:46.464857 21738 catalog_manager.cc:1347] Generated new cluster ID: 3bc6395fea27470bb27e4762d37bdb7f
I20260929 15:56:46.464998 21738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:56:46.475260 21738 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:56:46.475675 21738 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:56:46.484318 21738 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9022992314e44544b8a90a44a481aec7: Generated new TSK 0
I20260929 15:56:46.485100 21738 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:56:46.496811 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 21653
W20260929 15:56:46.497896 21715 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.189:50591
W20260929 15:56:46.498715 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 21653: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:56:46.498854 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 21653
2026-09-29T15:56:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1911 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:56:46 dist-test-slave-r0ck krb5kdc[21751](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:56:46 dist-test-slave-r0ck krb5kdc[21751](info): set up 2 sockets
Sep 29 15:56:46 dist-test-slave-r0ck krb5kdc[21751](info): commencing operation
krb5kdc: starting...
W20260929 15:56:47.597929 388 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.009s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:56:47 dist-test-slave-r0ck krb5kdc[21751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:56:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:56:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 15:56:47.861675 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36205
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:56:48.189947 21764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:56:48.190235 21764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:56:48.190387 21764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:56:48.197955 21764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:56:48.198084 21764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:56:48.198199 21764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:56:48.198285 21764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:56:48.198350 21764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:56:48.208542 21764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36205
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:56:48.209894 21764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:56:48.211164 21764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:56:48.223101 21769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:48.223201 21772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:48.223294 21770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:56:48.224385 21764 server_base.cc:1034] running on GCE node
I20260929 15:56:48.225173 21764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:56:48.226712 21764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:56:48.227973 21764 hybrid_clock.cc:648] HybridClock initialized: now 1790697408227901 us; error 63 us; skew 500 ppm
I20260929 15:56:48.228408 21764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:56:48 dist-test-slave-r0ck krb5kdc[21751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:56:48.231675 21764 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 15:56:48.233119 21764 webserver.cc:460] Webserver started at http://127.0.0.1:32961/ using document root <none> and password file <none>
I20260929 15:56:48.233848 21764 fs_manager.cc:359] Metadata directory not provided
I20260929 15:56:48.233950 21764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:56:48.234210 21764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:56:48.237221 21764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "99da9e95b6fd426abbcb3fc8c136cb51"
format_stamp: "Formatted at 2026-09-29 15:56:48 on dist-test-slave-r0ck"
server_key: "93e3cd8c9a8a0e98552300ebabafa50f"
server_key_iv: "0ea6122bd16990cad8d3012da96cb861"
server_key_version: "encryptionkey@0"
I20260929 15:56:48.237999 21764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "99da9e95b6fd426abbcb3fc8c136cb51"
format_stamp: "Formatted at 2026-09-29 15:56:48 on dist-test-slave-r0ck"
server_key: "93e3cd8c9a8a0e98552300ebabafa50f"
server_key_iv: "0ea6122bd16990cad8d3012da96cb861"
server_key_version: "encryptionkey@0"
I20260929 15:56:48.243872 21764 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 15:56:48.247921 21779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:48.249425 21764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 15:56:48.249585 21764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "99da9e95b6fd426abbcb3fc8c136cb51"
format_stamp: "Formatted at 2026-09-29 15:56:48 on dist-test-slave-r0ck"
server_key: "93e3cd8c9a8a0e98552300ebabafa50f"
server_key_iv: "0ea6122bd16990cad8d3012da96cb861"
server_key_version: "encryptionkey@0"
I20260929 15:56:48.249778 21764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:56:48.283001 21764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:56:48.287427 21764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:56:48.287683 21764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:56:48.306725 21764 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36205
I20260929 15:56:48.306756 21830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36205 every 8 connection(s)
I20260929 15:56:48.308123 21764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:56:48.311859 21831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:56:48.317408 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 21764
I20260929 15:56:48.317617 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:56:48.318061 388 external_mini_cluster.cc:1428] Setting key b9c9e7a6b0a024b27f092ac181858f25
I20260929 15:56:48.323047 21831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99da9e95b6fd426abbcb3fc8c136cb51: Bootstrap starting.
I20260929 15:56:48.326788 21831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99da9e95b6fd426abbcb3fc8c136cb51: Neither blocks nor log segments found. Creating new log.
I20260929 15:56:48.327756 21831 log.cc:824] T 00000000000000000000000000000000 P 99da9e95b6fd426abbcb3fc8c136cb51: Log is configured to *not* fsync() on all Append() calls
I20260929 15:56:48.330714 21831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99da9e95b6fd426abbcb3fc8c136cb51: No bootstrap required, opened a new log
Sep 29 15:56:48 dist-test-slave-r0ck krb5kdc[21751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 15:56:48.338938 21831 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99da9e95b6fd426abbcb3fc8c136cb51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99da9e95b6fd426abbcb3fc8c136cb51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36205 } }
I20260929 15:56:48.339356 21831 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99da9e95b6fd426abbcb3fc8c136cb51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:56:48.339498 21831 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99da9e95b6fd426abbcb3fc8c136cb51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99da9e95b6fd426abbcb3fc8c136cb51, State: Initialized, Role: FOLLOWER
I20260929 15:56:48.340188 21831 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99da9e95b6fd426abbcb3fc8c136cb51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99da9e95b6fd426abbcb3fc8c136cb51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36205 } }
I20260929 15:56:48.340456 21831 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99da9e95b6fd426abbcb3fc8c136cb51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:56:48.340615 21831 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99da9e95b6fd426abbcb3fc8c136cb51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:56:48.340806 21831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99da9e95b6fd426abbcb3fc8c136cb51 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:56:48.341177 21834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:48.319979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.189:34913 (local address 127.0.0.1:36205)
0929 15:56:48.320826 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:48.320837 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:48.320882 (+ 45us) server_negotiation.cc:407] Connection header received
0929 15:56:48.322874 (+ 1992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:48.322899 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:48.323370 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:48.323697 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:48.325573 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:48.326840 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:48.328202 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:48.328898 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:48.329085 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:48.333193 (+ 4108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:48.333200 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:48.333221 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:56:48.333268 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:48.336569 (+ 3301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:48.337455 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:48.337466 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:48.337472 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:48.337655 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:48.338172 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:48.338175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:48.338177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:48.338753 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:48.339082 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:48.339599 (+ 517us) server_negotiation.cc:299] Negotiation successful
0929 15:56:48.340002 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":398,"threads_started":1}
I20260929 15:56:48.343266 21831 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99da9e95b6fd426abbcb3fc8c136cb51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99da9e95b6fd426abbcb3fc8c136cb51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36205 } }
I20260929 15:56:48.343786 21831 leader_election.cc:302] T 00000000000000000000000000000000 P 99da9e95b6fd426abbcb3fc8c136cb51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99da9e95b6fd426abbcb3fc8c136cb51; no voters:
I20260929 15:56:48.344470 21831 leader_election.cc:288] T 00000000000000000000000000000000 P 99da9e95b6fd426abbcb3fc8c136cb51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:56:48.344568 21836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99da9e95b6fd426abbcb3fc8c136cb51 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:56:48.345080 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:56:48.347083 21831 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99da9e95b6fd426abbcb3fc8c136cb51 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:56:48.349434 21836 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99da9e95b6fd426abbcb3fc8c136cb51 [term 1 LEADER]: Becoming Leader. State: Replica: 99da9e95b6fd426abbcb3fc8c136cb51, State: Running, Role: LEADER
I20260929 15:56:48.350970 21836 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99da9e95b6fd426abbcb3fc8c136cb51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99da9e95b6fd426abbcb3fc8c136cb51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36205 } }
I20260929 15:56:48.354578 21837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99da9e95b6fd426abbcb3fc8c136cb51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99da9e95b6fd426abbcb3fc8c136cb51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99da9e95b6fd426abbcb3fc8c136cb51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36205 } } }
I20260929 15:56:48.355015 21837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99da9e95b6fd426abbcb3fc8c136cb51 [sys.catalog]: This master's current role is: LEADER
I20260929 15:56:48.355116 21838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99da9e95b6fd426abbcb3fc8c136cb51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99da9e95b6fd426abbcb3fc8c136cb51. Latest consensus state: current_term: 1 leader_uuid: "99da9e95b6fd426abbcb3fc8c136cb51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99da9e95b6fd426abbcb3fc8c136cb51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36205 } } }
I20260929 15:56:48.355415 21838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99da9e95b6fd426abbcb3fc8c136cb51 [sys.catalog]: This master's current role is: LEADER
I20260929 15:56:48.360693 21850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:56:48.366006 21850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:56:48.369850 21834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:48.349859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.189:52683 (local address 127.0.0.1:36205)
0929 15:56:48.350102 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:48.350112 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:48.350378 (+ 266us) server_negotiation.cc:407] Connection header received
0929 15:56:48.350587 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:48.350599 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:48.350663 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:48.350769 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:48.352331 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:48.353449 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:48.355466 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:48.356154 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:48.356310 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:48.359972 (+ 3662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:48.359978 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:48.359983 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:56:48.360021 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:48.364195 (+ 4174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:48.365499 (+ 1304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:48.365508 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:48.365514 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:48.365755 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:48.366463 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:48.366471 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:48.366477 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:48.367006 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:48.367323 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:48.369012 (+ 1689us) server_negotiation.cc:299] Negotiation successful
0929 15:56:48.369275 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 15:56:48.379761 21850 catalog_manager.cc:1347] Generated new cluster ID: 3cd03bf702df493d80bcfbcf2145227b
I20260929 15:56:48.379885 21850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:56:48.397188 21850 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:56:48.398501 21850 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:56:48.418905 21850 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99da9e95b6fd426abbcb3fc8c136cb51: Generated new TSK 0
I20260929 15:56:48.419725 21850 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:56:48.428050 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 21764
I20260929 15:56:48.429152 21826 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:56:48.521893 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 21764
W20260929 15:56:48.538887 21749 connection.cc:535] client connection to 127.0.0.1:36205 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36205): Connection reset by peer (error 104)
2026-09-29T15:56:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2023 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:56:48 dist-test-slave-r0ck krb5kdc[21863](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:56:48 dist-test-slave-r0ck krb5kdc[21863](info): set up 2 sockets
Sep 29 15:56:48 dist-test-slave-r0ck krb5kdc[21863](info): commencing operation
krb5kdc: starting...
W20260929 15:56:49.629787 388 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:56:49 dist-test-slave-r0ck krb5kdc[21863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:56:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:56:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 15:56:49.886140 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35863
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:56:50.206964 21876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:56:50.207305 21876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:56:50.207472 21876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:56:50.215119 21876 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:56:50.215270 21876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:56:50.215355 21876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:56:50.215441 21876 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:56:50.215507 21876 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:56:50.225147 21876 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35863
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:56:50.226452 21876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:56:50.227725 21876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:56:50.239732 21881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:50.239679 21882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:50.239645 21884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:56:50.240661 21876 server_base.cc:1034] running on GCE node
I20260929 15:56:50.241482 21876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:56:50.242794 21876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:56:50.243999 21876 hybrid_clock.cc:648] HybridClock initialized: now 1790697410243954 us; error 39 us; skew 500 ppm
I20260929 15:56:50.244339 21876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:56:50 dist-test-slave-r0ck krb5kdc[21863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:56:50.247386 21876 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 15:56:50.248935 21876 webserver.cc:460] Webserver started at http://127.0.0.1:46529/ using document root <none> and password file <none>
I20260929 15:56:50.249626 21876 fs_manager.cc:359] Metadata directory not provided
I20260929 15:56:50.249727 21876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:56:50.249965 21876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:56:50.252736 21876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "53271c4caaac4c21a78ce65bda6be0be"
format_stamp: "Formatted at 2026-09-29 15:56:50 on dist-test-slave-r0ck"
server_key: "abb32d992d903b7a2cb699cfaead0404"
server_key_iv: "92df136584aefd504511d6831e2860de"
server_key_version: "encryptionkey@0"
I20260929 15:56:50.253475 21876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "53271c4caaac4c21a78ce65bda6be0be"
format_stamp: "Formatted at 2026-09-29 15:56:50 on dist-test-slave-r0ck"
server_key: "abb32d992d903b7a2cb699cfaead0404"
server_key_iv: "92df136584aefd504511d6831e2860de"
server_key_version: "encryptionkey@0"
I20260929 15:56:50.258518 21876 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 15:56:50.262435 21891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:50.263809 21876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:56:50.263984 21876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "53271c4caaac4c21a78ce65bda6be0be"
format_stamp: "Formatted at 2026-09-29 15:56:50 on dist-test-slave-r0ck"
server_key: "abb32d992d903b7a2cb699cfaead0404"
server_key_iv: "92df136584aefd504511d6831e2860de"
server_key_version: "encryptionkey@0"
I20260929 15:56:50.264179 21876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:56:50.292982 21876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:56:50.297173 21876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:56:50.297500 21876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:56:50.316352 21876 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35863
I20260929 15:56:50.316394 21942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35863 every 8 connection(s)
I20260929 15:56:50.317790 21876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:56:50.320068 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 21876
I20260929 15:56:50.320290 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:56:50.320765 388 external_mini_cluster.cc:1428] Setting key 819907b307ba1150069cb3e584872e2e
I20260929 15:56:50.321967 21943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:56:50.336308 21946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:50.322624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.189:49437 (local address 127.0.0.1:35863)
0929 15:56:50.323392 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:50.323406 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:50.323554 (+ 148us) server_negotiation.cc:407] Connection header received
0929 15:56:50.325641 (+ 2087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:50.325672 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:50.326234 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:50.326722 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:50.327346 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:50.329479 (+ 2133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:50.330925 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:50.331977 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:50.332171 (+ 194us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:50.333453 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:50.333469 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:50.333488 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:50.334244 (+ 756us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:50.334339 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:50.334376 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:50.334537 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:50.334822 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 15:56:50.335009 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":316,"threads_started":1}
I20260929 15:56:50.337705 21943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53271c4caaac4c21a78ce65bda6be0be: Bootstrap starting.
I20260929 15:56:50.340270 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:56:50.341019 21943 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53271c4caaac4c21a78ce65bda6be0be: Neither blocks nor log segments found. Creating new log.
I20260929 15:56:50.342062 21943 log.cc:824] T 00000000000000000000000000000000 P 53271c4caaac4c21a78ce65bda6be0be: Log is configured to *not* fsync() on all Append() calls
I20260929 15:56:50.345294 21943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53271c4caaac4c21a78ce65bda6be0be: No bootstrap required, opened a new log
I20260929 15:56:50.351617 21946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:50.344777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.189:55641 (local address 127.0.0.1:35863)
0929 15:56:50.344983 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:50.344990 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:50.345289 (+ 299us) server_negotiation.cc:407] Connection header received
0929 15:56:50.345531 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:50.345535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:50.345611 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:50.345691 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:50.346293 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:50.347082 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:50.348992 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:50.349518 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:50.349636 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:50.350820 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:50.350828 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:50.350831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:50.350847 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:50.350883 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:50.350897 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:50.350986 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:50.351198 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 15:56:50.351268 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 15:56:50.354633 21943 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53271c4caaac4c21a78ce65bda6be0be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53271c4caaac4c21a78ce65bda6be0be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35863 } }
I20260929 15:56:50.354964 21943 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53271c4caaac4c21a78ce65bda6be0be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:56:50.355106 21943 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53271c4caaac4c21a78ce65bda6be0be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53271c4caaac4c21a78ce65bda6be0be, State: Initialized, Role: FOLLOWER
I20260929 15:56:50.355669 21943 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53271c4caaac4c21a78ce65bda6be0be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53271c4caaac4c21a78ce65bda6be0be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35863 } }
I20260929 15:56:50.355862 21943 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53271c4caaac4c21a78ce65bda6be0be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:56:50.355960 21943 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53271c4caaac4c21a78ce65bda6be0be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:56:50.356073 21943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53271c4caaac4c21a78ce65bda6be0be [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:56:50.358523 21943 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53271c4caaac4c21a78ce65bda6be0be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53271c4caaac4c21a78ce65bda6be0be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35863 } }
I20260929 15:56:50.358950 21943 leader_election.cc:302] T 00000000000000000000000000000000 P 53271c4caaac4c21a78ce65bda6be0be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53271c4caaac4c21a78ce65bda6be0be; no voters:
I20260929 15:56:50.359498 21943 leader_election.cc:288] T 00000000000000000000000000000000 P 53271c4caaac4c21a78ce65bda6be0be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:56:50.359628 21953 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53271c4caaac4c21a78ce65bda6be0be [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:56:50.360414 21953 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53271c4caaac4c21a78ce65bda6be0be [term 1 LEADER]: Becoming Leader. State: Replica: 53271c4caaac4c21a78ce65bda6be0be, State: Running, Role: LEADER
I20260929 15:56:50.360980 21953 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53271c4caaac4c21a78ce65bda6be0be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53271c4caaac4c21a78ce65bda6be0be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35863 } }
I20260929 15:56:50.361706 21943 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53271c4caaac4c21a78ce65bda6be0be [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:56:50.363833 21954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53271c4caaac4c21a78ce65bda6be0be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53271c4caaac4c21a78ce65bda6be0be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53271c4caaac4c21a78ce65bda6be0be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35863 } } }
I20260929 15:56:50.364204 21954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53271c4caaac4c21a78ce65bda6be0be [sys.catalog]: This master's current role is: LEADER
I20260929 15:56:50.367165 21960 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:56:50.367369 21955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53271c4caaac4c21a78ce65bda6be0be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53271c4caaac4c21a78ce65bda6be0be. Latest consensus state: current_term: 1 leader_uuid: "53271c4caaac4c21a78ce65bda6be0be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53271c4caaac4c21a78ce65bda6be0be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35863 } } }
I20260929 15:56:50.367688 21955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53271c4caaac4c21a78ce65bda6be0be [sys.catalog]: This master's current role is: LEADER
I20260929 15:56:50.371352 21960 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:56:50.380030 21960 catalog_manager.cc:1347] Generated new cluster ID: b78c769bfd134d0095b71deff8b5d24d
I20260929 15:56:50.380148 21960 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:56:50.396751 21960 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:56:50.398072 21960 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:56:50.405748 21960 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53271c4caaac4c21a78ce65bda6be0be: Generated new TSK 0
I20260929 15:56:50.406575 21960 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:56:50.418112 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 21876
W20260929 15:56:50.419250 21938 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.189:49437
W20260929 15:56:50.420076 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 21876: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:56:50.420217 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 21876
W20260929 15:56:50.437088 21861 connection.cc:535] client connection to 127.0.0.1:35863 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35863): Connection reset by peer (error 104)
2026-09-29T15:56:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1902 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:56:50 dist-test-slave-r0ck krb5kdc[21974](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:56:50 dist-test-slave-r0ck krb5kdc[21974](info): set up 2 sockets
Sep 29 15:56:50 dist-test-slave-r0ck krb5kdc[21974](info): commencing operation
krb5kdc: starting...
W20260929 15:56:51.545760 388 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:56:51 dist-test-slave-r0ck krb5kdc[21974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:56:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:56:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 15:56:51.792865 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40947
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:56:52.116186 21987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:56:52.116472 21987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:56:52.116634 21987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:56:52.124081 21987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:56:52.124223 21987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:56:52.124372 21987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:56:52.124490 21987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:56:52.124564 21987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:56:52.134368 21987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40947
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697075657932-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:56:52.135608 21987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:56:52.136842 21987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:56:52.147995 21992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:52.148041 21993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:52.148124 21995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:56:52.149021 21987 server_base.cc:1034] running on GCE node
I20260929 15:56:52.149784 21987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:56:52.151032 21987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:56:52.152261 21987 hybrid_clock.cc:648] HybridClock initialized: now 1790697412152194 us; error 50 us; skew 500 ppm
I20260929 15:56:52.152648 21987 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:56:52 dist-test-slave-r0ck krb5kdc[21974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:56:52.160248 21987 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 15:56:52.161737 21987 webserver.cc:460] Webserver started at http://127.0.0.1:34097/ using document root <none> and password file <none>
I20260929 15:56:52.162405 21987 fs_manager.cc:359] Metadata directory not provided
I20260929 15:56:52.162535 21987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:56:52.162842 21987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:56:52.165745 21987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "93debc1a36cf4c9ead0f3bdaf0fe11e1"
format_stamp: "Formatted at 2026-09-29 15:56:52 on dist-test-slave-r0ck"
server_key: "fe8c38ea16ab7465d0766bea8db68b78"
server_key_iv: "f19c6bee0ca8fae172e81c331897d6a6"
server_key_version: "encryptionkey@0"
I20260929 15:56:52.166491 21987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "93debc1a36cf4c9ead0f3bdaf0fe11e1"
format_stamp: "Formatted at 2026-09-29 15:56:52 on dist-test-slave-r0ck"
server_key: "fe8c38ea16ab7465d0766bea8db68b78"
server_key_iv: "f19c6bee0ca8fae172e81c331897d6a6"
server_key_version: "encryptionkey@0"
I20260929 15:56:52.171598 21987 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 15:56:52.174921 22002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:52.176220 21987 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 15:56:52.176402 21987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "93debc1a36cf4c9ead0f3bdaf0fe11e1"
format_stamp: "Formatted at 2026-09-29 15:56:52 on dist-test-slave-r0ck"
server_key: "fe8c38ea16ab7465d0766bea8db68b78"
server_key_iv: "f19c6bee0ca8fae172e81c331897d6a6"
server_key_version: "encryptionkey@0"
I20260929 15:56:52.176589 21987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:56:52.182463 21987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:56:52.186168 21987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:56:52.186446 21987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:56:52.204309 21987 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40947
I20260929 15:56:52.204380 22053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40947 every 8 connection(s)
I20260929 15:56:52.205689 21987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:56:52.209139 22054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:56:52.209724 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 21987
I20260929 15:56:52.209908 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:56:52.210337 388 external_mini_cluster.cc:1428] Setting key d4a612c03c815e4ffa5c41c0a79ca152
I20260929 15:56:52.219205 22057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:52.212160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.189:33045 (local address 127.0.0.1:40947)
0929 15:56:52.212943 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:52.212957 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:52.213005 (+ 48us) server_negotiation.cc:407] Connection header received
0929 15:56:52.215579 (+ 2574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:52.215605 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:52.216069 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:52.216496 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:52.217430 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:52.217438 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:52.217453 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:52.217471 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:52.217531 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:52.217554 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:52.217770 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:52.217906 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 15:56:52.218079 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":294,"threads_started":1}
I20260929 15:56:52.222760 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:56:52.223361 22054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93debc1a36cf4c9ead0f3bdaf0fe11e1: Bootstrap starting.
I20260929 15:56:52.226248 22054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93debc1a36cf4c9ead0f3bdaf0fe11e1: Neither blocks nor log segments found. Creating new log.
I20260929 15:56:52.227452 22054 log.cc:824] T 00000000000000000000000000000000 P 93debc1a36cf4c9ead0f3bdaf0fe11e1: Log is configured to *not* fsync() on all Append() calls
I20260929 15:56:52.229651 22057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:52.226995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.189:48377 (local address 127.0.0.1:40947)
0929 15:56:52.227208 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:52.227216 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:52.227645 (+ 429us) server_negotiation.cc:407] Connection header received
0929 15:56:52.227793 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:52.227796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:52.227841 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:52.227916 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:52.228965 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:52.228968 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:52.228971 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:52.228984 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:52.229022 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:52.229036 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:52.229160 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:52.229277 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 15:56:52.229320 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 15:56:52.230487 22054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93debc1a36cf4c9ead0f3bdaf0fe11e1: No bootstrap required, opened a new log
I20260929 15:56:52.237315 22054 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93debc1a36cf4c9ead0f3bdaf0fe11e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93debc1a36cf4c9ead0f3bdaf0fe11e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40947 } }
I20260929 15:56:52.237746 22054 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93debc1a36cf4c9ead0f3bdaf0fe11e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:56:52.237852 22054 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93debc1a36cf4c9ead0f3bdaf0fe11e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93debc1a36cf4c9ead0f3bdaf0fe11e1, State: Initialized, Role: FOLLOWER
I20260929 15:56:52.238478 22054 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93debc1a36cf4c9ead0f3bdaf0fe11e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93debc1a36cf4c9ead0f3bdaf0fe11e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40947 } }
I20260929 15:56:52.238744 22054 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93debc1a36cf4c9ead0f3bdaf0fe11e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:56:52.238925 22054 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93debc1a36cf4c9ead0f3bdaf0fe11e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:56:52.239104 22054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93debc1a36cf4c9ead0f3bdaf0fe11e1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:56:52.241169 22054 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93debc1a36cf4c9ead0f3bdaf0fe11e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93debc1a36cf4c9ead0f3bdaf0fe11e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40947 } }
I20260929 15:56:52.241667 22054 leader_election.cc:302] T 00000000000000000000000000000000 P 93debc1a36cf4c9ead0f3bdaf0fe11e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93debc1a36cf4c9ead0f3bdaf0fe11e1; no voters:
I20260929 15:56:52.242241 22054 leader_election.cc:288] T 00000000000000000000000000000000 P 93debc1a36cf4c9ead0f3bdaf0fe11e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:56:52.242417 22064 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93debc1a36cf4c9ead0f3bdaf0fe11e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:56:52.243058 22064 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93debc1a36cf4c9ead0f3bdaf0fe11e1 [term 1 LEADER]: Becoming Leader. State: Replica: 93debc1a36cf4c9ead0f3bdaf0fe11e1, State: Running, Role: LEADER
I20260929 15:56:52.243577 22064 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93debc1a36cf4c9ead0f3bdaf0fe11e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93debc1a36cf4c9ead0f3bdaf0fe11e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40947 } }
I20260929 15:56:52.244177 22054 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93debc1a36cf4c9ead0f3bdaf0fe11e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:56:52.246647 22066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93debc1a36cf4c9ead0f3bdaf0fe11e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93debc1a36cf4c9ead0f3bdaf0fe11e1. Latest consensus state: current_term: 1 leader_uuid: "93debc1a36cf4c9ead0f3bdaf0fe11e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93debc1a36cf4c9ead0f3bdaf0fe11e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40947 } } }
I20260929 15:56:52.247123 22066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93debc1a36cf4c9ead0f3bdaf0fe11e1 [sys.catalog]: This master's current role is: LEADER
I20260929 15:56:52.246623 22065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93debc1a36cf4c9ead0f3bdaf0fe11e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93debc1a36cf4c9ead0f3bdaf0fe11e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93debc1a36cf4c9ead0f3bdaf0fe11e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40947 } } }
I20260929 15:56:52.247793 22065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93debc1a36cf4c9ead0f3bdaf0fe11e1 [sys.catalog]: This master's current role is: LEADER
I20260929 15:56:52.254135 22071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:56:52.259001 22071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:56:52.269304 22071 catalog_manager.cc:1347] Generated new cluster ID: 88eac7c768744fd4ac84944f6cbb908f
I20260929 15:56:52.269454 22071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:56:52.278084 22071 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:56:52.278544 22071 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:56:52.291844 22071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93debc1a36cf4c9ead0f3bdaf0fe11e1: Generated new TSK 0
I20260929 15:56:52.292886 22071 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:56:52.298472 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 21987
W20260929 15:56:52.299741 22049 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.189:33045
W20260929 15:56:52.300426 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 21987: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:56:52.300570 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 21987
2026-09-29T15:56:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1875 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:56:52 dist-test-slave-r0ck krb5kdc[22085](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:56:52 dist-test-slave-r0ck krb5kdc[22085](info): set up 2 sockets
Sep 29 15:56:52 dist-test-slave-r0ck krb5kdc[22085](info): commencing operation
krb5kdc: starting...
W20260929 15:56:53.397894 388 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.006s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:56:53 dist-test-slave-r0ck krb5kdc[22085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:56:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:56:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:56:53.668735 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36403
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:56:54.006018 22098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:56:54.006335 22098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:56:54.006433 22098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:56:54.014053 22098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:56:54.014186 22098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:56:54.014235 22098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:56:54.014288 22098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:56:54.014335 22098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:56:54.024765 22098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36403
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:56:54.026093 22098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:56:54.027333 22098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:56:54.039871 22103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:54.039922 22106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:54.039919 22104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:56:54.041029 22098 server_base.cc:1034] running on GCE node
I20260929 15:56:54.041862 22098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:56:54.043341 22098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:56:54.044574 22098 hybrid_clock.cc:648] HybridClock initialized: now 1790697414044540 us; error 62 us; skew 500 ppm
I20260929 15:56:54.044945 22098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:56:54 dist-test-slave-r0ck krb5kdc[22085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:56:54.048137 22098 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:56:54.049839 22098 webserver.cc:460] Webserver started at http://127.0.97.62:42393/ using document root <none> and password file <none>
I20260929 15:56:54.050540 22098 fs_manager.cc:359] Metadata directory not provided
I20260929 15:56:54.050685 22098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:56:54.050993 22098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:56:54.053984 22098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "773f5dd82edf4b5a8543140baca88d03"
format_stamp: "Formatted at 2026-09-29 15:56:54 on dist-test-slave-r0ck"
server_key: "bcc555e0530a07e060b97906a170c7b8"
server_key_iv: "5b7b0d1d3e971db2451504958bb3d375"
server_key_version: "encryptionkey@0"
I20260929 15:56:54.054733 22098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "773f5dd82edf4b5a8543140baca88d03"
format_stamp: "Formatted at 2026-09-29 15:56:54 on dist-test-slave-r0ck"
server_key: "bcc555e0530a07e060b97906a170c7b8"
server_key_iv: "5b7b0d1d3e971db2451504958bb3d375"
server_key_version: "encryptionkey@0"
I20260929 15:56:54.059787 22098 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 15:56:54.063583 22113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:54.065034 22098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 15:56:54.065239 22098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "773f5dd82edf4b5a8543140baca88d03"
format_stamp: "Formatted at 2026-09-29 15:56:54 on dist-test-slave-r0ck"
server_key: "bcc555e0530a07e060b97906a170c7b8"
server_key_iv: "5b7b0d1d3e971db2451504958bb3d375"
server_key_version: "encryptionkey@0"
I20260929 15:56:54.065505 22098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:56:54.093135 22098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:56:54.097296 22098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:56:54.097623 22098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:56:54.116684 22098 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36403
I20260929 15:56:54.116736 22164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36403 every 8 connection(s)
I20260929 15:56:54.118096 22098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:56:54.122195 22165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:56:54.123914 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 22098
I20260929 15:56:54.124121 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:56:54.124691 388 external_mini_cluster.cc:1428] Setting key 96ef7fca79202dca4a93532c8b5aed92
I20260929 15:56:54.136234 22165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 773f5dd82edf4b5a8543140baca88d03: Bootstrap starting.
Sep 29 15:56:54 dist-test-slave-r0ck krb5kdc[22085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:56:54.140061 22165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 773f5dd82edf4b5a8543140baca88d03: Neither blocks nor log segments found. Creating new log.
I20260929 15:56:54.141494 22165 log.cc:824] T 00000000000000000000000000000000 P 773f5dd82edf4b5a8543140baca88d03: Log is configured to *not* fsync() on all Append() calls
I20260929 15:56:54.144814 22165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 773f5dd82edf4b5a8543140baca88d03: No bootstrap required, opened a new log
I20260929 15:56:54.147954 22168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:54.126724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46683 (local address 127.0.97.62:36403)
0929 15:56:54.127645 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:54.127657 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:54.127695 (+ 38us) server_negotiation.cc:407] Connection header received
0929 15:56:54.129308 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:54.129332 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:54.129783 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:54.130178 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:54.131645 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:54.132983 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:54.134310 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:54.134931 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:54.135115 (+ 184us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:54.139867 (+ 4752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:54.139875 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:54.139888 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:56:54.139928 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:54.143623 (+ 3695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:54.144635 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:54.144642 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:54.144648 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:54.144820 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:54.145329 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:54.145332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:54.145334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:54.145868 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:54.146113 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:54.146613 (+ 500us) server_negotiation.cc:299] Negotiation successful
0929 15:56:54.146908 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":714,"thread_start_us":451,"threads_started":1}
I20260929 15:56:54.152074 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:56:54.155030 22165 raft_consensus.cc:348] T 00000000000000000000000000000000 P 773f5dd82edf4b5a8543140baca88d03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "773f5dd82edf4b5a8543140baca88d03" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36403 } }
I20260929 15:56:54.155516 22165 raft_consensus.cc:374] T 00000000000000000000000000000000 P 773f5dd82edf4b5a8543140baca88d03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:56:54.155675 22165 raft_consensus.cc:729] T 00000000000000000000000000000000 P 773f5dd82edf4b5a8543140baca88d03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 773f5dd82edf4b5a8543140baca88d03, State: Initialized, Role: FOLLOWER
I20260929 15:56:54.156453 22165 consensus_queue.cc:260] T 00000000000000000000000000000000 P 773f5dd82edf4b5a8543140baca88d03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "773f5dd82edf4b5a8543140baca88d03" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36403 } }
I20260929 15:56:54.156744 22165 raft_consensus.cc:388] T 00000000000000000000000000000000 P 773f5dd82edf4b5a8543140baca88d03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:56:54.156890 22165 raft_consensus.cc:482] T 00000000000000000000000000000000 P 773f5dd82edf4b5a8543140baca88d03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:56:54.157060 22165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 773f5dd82edf4b5a8543140baca88d03 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:56:54.159718 22165 raft_consensus.cc:504] T 00000000000000000000000000000000 P 773f5dd82edf4b5a8543140baca88d03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "773f5dd82edf4b5a8543140baca88d03" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36403 } }
I20260929 15:56:54.160315 22165 leader_election.cc:302] T 00000000000000000000000000000000 P 773f5dd82edf4b5a8543140baca88d03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 773f5dd82edf4b5a8543140baca88d03; no voters:
I20260929 15:56:54.160946 22165 leader_election.cc:288] T 00000000000000000000000000000000 P 773f5dd82edf4b5a8543140baca88d03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:56:54.161068 22175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 773f5dd82edf4b5a8543140baca88d03 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:56:54.162966 22165 sys_catalog.cc:564] T 00000000000000000000000000000000 P 773f5dd82edf4b5a8543140baca88d03 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:56:54.163780 22175 raft_consensus.cc:686] T 00000000000000000000000000000000 P 773f5dd82edf4b5a8543140baca88d03 [term 1 LEADER]: Becoming Leader. State: Replica: 773f5dd82edf4b5a8543140baca88d03, State: Running, Role: LEADER
I20260929 15:56:54.164405 22175 consensus_queue.cc:237] T 00000000000000000000000000000000 P 773f5dd82edf4b5a8543140baca88d03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "773f5dd82edf4b5a8543140baca88d03" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36403 } }
I20260929 15:56:54.167337 22176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 773f5dd82edf4b5a8543140baca88d03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "773f5dd82edf4b5a8543140baca88d03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "773f5dd82edf4b5a8543140baca88d03" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36403 } } }
I20260929 15:56:54.168008 22176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 773f5dd82edf4b5a8543140baca88d03 [sys.catalog]: This master's current role is: LEADER
I20260929 15:56:54.167856 22177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 773f5dd82edf4b5a8543140baca88d03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 773f5dd82edf4b5a8543140baca88d03. Latest consensus state: current_term: 1 leader_uuid: "773f5dd82edf4b5a8543140baca88d03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "773f5dd82edf4b5a8543140baca88d03" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36403 } } }
I20260929 15:56:54.168162 22177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 773f5dd82edf4b5a8543140baca88d03 [sys.catalog]: This master's current role is: LEADER
I20260929 15:56:54.173926 22183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:56:54.175420 22168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:54.157545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38987 (local address 127.0.97.62:36403)
0929 15:56:54.157733 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:54.157739 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:54.158256 (+ 517us) server_negotiation.cc:407] Connection header received
0929 15:56:54.158376 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:54.158382 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:54.158468 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:54.158590 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:54.159920 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:54.161080 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:54.163352 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:54.164187 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:54.164357 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:54.167443 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:54.167452 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:54.167460 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:56:54.167515 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:54.172178 (+ 4663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:54.173041 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:54.173046 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:54.173049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:54.173217 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:56:54.173767 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:56:54.173772 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:56:54.173775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:56:54.174177 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:54.174437 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:54.174862 (+ 425us) server_negotiation.cc:299] Negotiation successful
0929 15:56:54.175039 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 15:56:54.180315 22183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:56:54.189436 22183 catalog_manager.cc:1347] Generated new cluster ID: a380e737d0c54ca699507b38bfa881c0
I20260929 15:56:54.189543 22183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:56:54.214785 22183 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:56:54.216677 22183 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:56:54.242671 22183 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 773f5dd82edf4b5a8543140baca88d03: Generated new TSK 0
I20260929 15:56:54.243541 22183 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:56:54.258132 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 22098
I20260929 15:56:54.259763 22160 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:56:54.353595 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 22098
W20260929 15:56:54.371188 22083 connection.cc:535] client connection to 127.0.97.62:36403 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:36403): Connection reset by peer (error 104)
2026-09-29T15:56:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2054 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:56:54 dist-test-slave-r0ck krb5kdc[22197](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:56:54 dist-test-slave-r0ck krb5kdc[22197](info): set up 2 sockets
Sep 29 15:56:54 dist-test-slave-r0ck krb5kdc[22197](info): commencing operation
krb5kdc: starting...
W20260929 15:56:55.449747 388 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.011s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:56:55 dist-test-slave-r0ck krb5kdc[22197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:56:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:56:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:56:55.700860 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38141
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:56:56.033406 22210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:56:56.033715 22210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:56:56.033872 22210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:56:56.041091 22210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:56:56.041199 22210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:56:56.041311 22210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:56:56.041419 22210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:56:56.041486 22210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:56:56.051386 22210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38141
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:56:56.052697 22210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:56:56.054069 22210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:56:56.065984 22215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:56.065907 22218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:56.065953 22216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:56:56.067728 22210 server_base.cc:1034] running on GCE node
I20260929 15:56:56.068567 22210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:56:56.070061 22210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:56:56.071280 22210 hybrid_clock.cc:648] HybridClock initialized: now 1790697416071215 us; error 54 us; skew 500 ppm
I20260929 15:56:56.071659 22210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:56:56 dist-test-slave-r0ck krb5kdc[22197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:56:56.074949 22210 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:56:56.076453 22210 webserver.cc:460] Webserver started at http://127.0.97.62:35023/ using document root <none> and password file <none>
I20260929 15:56:56.077133 22210 fs_manager.cc:359] Metadata directory not provided
I20260929 15:56:56.077265 22210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:56:56.077611 22210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:56:56.080528 22210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "5b9457089045412498976ffb1bb9dd7e"
format_stamp: "Formatted at 2026-09-29 15:56:56 on dist-test-slave-r0ck"
server_key: "fa0941a3d8f31630b1ec28b70696e451"
server_key_iv: "64ad0bc35f0cec3e55f96bedb86cb503"
server_key_version: "encryptionkey@0"
I20260929 15:56:56.081295 22210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "5b9457089045412498976ffb1bb9dd7e"
format_stamp: "Formatted at 2026-09-29 15:56:56 on dist-test-slave-r0ck"
server_key: "fa0941a3d8f31630b1ec28b70696e451"
server_key_iv: "64ad0bc35f0cec3e55f96bedb86cb503"
server_key_version: "encryptionkey@0"
I20260929 15:56:56.086496 22210 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 15:56:56.089970 22225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:56.091229 22210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 15:56:56.091423 22210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "5b9457089045412498976ffb1bb9dd7e"
format_stamp: "Formatted at 2026-09-29 15:56:56 on dist-test-slave-r0ck"
server_key: "fa0941a3d8f31630b1ec28b70696e451"
server_key_iv: "64ad0bc35f0cec3e55f96bedb86cb503"
server_key_version: "encryptionkey@0"
I20260929 15:56:56.091598 22210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:56:56.109864 22210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:56:56.114394 22210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:56:56.114668 22210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:56:56.132854 22210 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38141
I20260929 15:56:56.132884 22276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38141 every 8 connection(s)
I20260929 15:56:56.134285 22210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:56:56.134826 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 22210
I20260929 15:56:56.135037 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:56:56.135634 388 external_mini_cluster.cc:1428] Setting key d0236b89f2d93c1a9bc6029d2cbcce7b
I20260929 15:56:56.138645 22277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:56:56.149201 22280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:56.137712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33019 (local address 127.0.97.62:38141)
0929 15:56:56.138497 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:56.138508 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:56.138547 (+ 39us) server_negotiation.cc:407] Connection header received
0929 15:56:56.140175 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:56.140194 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:56.140549 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:56.140860 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:56.141442 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:56.142762 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:56.144590 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:56.145243 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:56.145473 (+ 230us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:56.146766 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:56.146777 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:56.146790 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:56.147372 (+ 582us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:56.147536 (+ 164us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:56.147572 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:56.147714 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:56.148043 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 15:56:56.148266 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":360,"threads_started":1}
I20260929 15:56:56.150763 22277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b9457089045412498976ffb1bb9dd7e: Bootstrap starting.
I20260929 15:56:56.152974 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:56:56.154071 22277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b9457089045412498976ffb1bb9dd7e: Neither blocks nor log segments found. Creating new log.
I20260929 15:56:56.155236 22277 log.cc:824] T 00000000000000000000000000000000 P 5b9457089045412498976ffb1bb9dd7e: Log is configured to *not* fsync() on all Append() calls
I20260929 15:56:56.158181 22277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b9457089045412498976ffb1bb9dd7e: No bootstrap required, opened a new log
I20260929 15:56:56.164948 22280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:56.158241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59239 (local address 127.0.97.62:38141)
0929 15:56:56.158402 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:56.158407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:56.158831 (+ 424us) server_negotiation.cc:407] Connection header received
0929 15:56:56.158973 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:56.158979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:56.159050 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:56.159157 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:56.159586 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:56.160788 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:56.162186 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:56:56.162889 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:56:56.163024 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:56:56.164097 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:56.164103 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:56.164110 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:56.164136 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:56.164230 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:56.164252 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:56.164381 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:56.164546 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 15:56:56.164613 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 15:56:56.167361 22277 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b9457089045412498976ffb1bb9dd7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b9457089045412498976ffb1bb9dd7e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38141 } }
I20260929 15:56:56.167737 22277 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b9457089045412498976ffb1bb9dd7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:56:56.167848 22277 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b9457089045412498976ffb1bb9dd7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b9457089045412498976ffb1bb9dd7e, State: Initialized, Role: FOLLOWER
I20260929 15:56:56.168464 22277 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b9457089045412498976ffb1bb9dd7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b9457089045412498976ffb1bb9dd7e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38141 } }
I20260929 15:56:56.168712 22277 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b9457089045412498976ffb1bb9dd7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:56:56.168848 22277 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b9457089045412498976ffb1bb9dd7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:56:56.169031 22277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b9457089045412498976ffb1bb9dd7e [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:56:56.171146 22277 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b9457089045412498976ffb1bb9dd7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b9457089045412498976ffb1bb9dd7e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38141 } }
I20260929 15:56:56.171612 22277 leader_election.cc:302] T 00000000000000000000000000000000 P 5b9457089045412498976ffb1bb9dd7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b9457089045412498976ffb1bb9dd7e; no voters:
I20260929 15:56:56.172278 22277 leader_election.cc:288] T 00000000000000000000000000000000 P 5b9457089045412498976ffb1bb9dd7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:56:56.172379 22287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b9457089045412498976ffb1bb9dd7e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:56:56.173326 22287 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b9457089045412498976ffb1bb9dd7e [term 1 LEADER]: Becoming Leader. State: Replica: 5b9457089045412498976ffb1bb9dd7e, State: Running, Role: LEADER
I20260929 15:56:56.173870 22287 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b9457089045412498976ffb1bb9dd7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b9457089045412498976ffb1bb9dd7e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38141 } }
I20260929 15:56:56.174340 22277 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b9457089045412498976ffb1bb9dd7e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:56:56.176774 22288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b9457089045412498976ffb1bb9dd7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b9457089045412498976ffb1bb9dd7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b9457089045412498976ffb1bb9dd7e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38141 } } }
I20260929 15:56:56.177220 22288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b9457089045412498976ffb1bb9dd7e [sys.catalog]: This master's current role is: LEADER
I20260929 15:56:56.177961 22289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b9457089045412498976ffb1bb9dd7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b9457089045412498976ffb1bb9dd7e. Latest consensus state: current_term: 1 leader_uuid: "5b9457089045412498976ffb1bb9dd7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b9457089045412498976ffb1bb9dd7e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38141 } } }
I20260929 15:56:56.178612 22289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b9457089045412498976ffb1bb9dd7e [sys.catalog]: This master's current role is: LEADER
I20260929 15:56:56.179397 22296 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:56:56.183663 22296 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:56:56.191725 22296 catalog_manager.cc:1347] Generated new cluster ID: ffedfbe0041c455faace0d49af6c6a27
I20260929 15:56:56.191831 22296 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:56:56.229996 22296 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:56:56.231266 22296 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:56:56.236809 22296 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b9457089045412498976ffb1bb9dd7e: Generated new TSK 0
I20260929 15:56:56.237787 22296 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:56:56.254683 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 22210
W20260929 15:56:56.255887 22272 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33019
W20260929 15:56:56.256603 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 22210: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:56:56.256752 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 22210
W20260929 15:56:56.273866 22195 connection.cc:535] client connection to 127.0.97.62:38141 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:38141): Connection reset by peer (error 104)
2026-09-29T15:56:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1901 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:56:56 dist-test-slave-r0ck krb5kdc[22308](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:56:56 dist-test-slave-r0ck krb5kdc[22308](info): set up 2 sockets
Sep 29 15:56:56 dist-test-slave-r0ck krb5kdc[22308](info): commencing operation
krb5kdc: starting...
W20260929 15:56:57.370386 388 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:56:57 dist-test-slave-r0ck krb5kdc[22308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:56:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:56:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:56:57.666234 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38291
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:56:58.010353 22321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:56:58.010658 22321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:56:58.010754 22321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:56:58.018512 22321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:56:58.018637 22321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:56:58.018684 22321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:56:58.018788 22321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:56:58.018864 22321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:56:58.029024 22321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38291
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:56:58.030331 22321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:56:58.031601 22321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:56:58.043358 22329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:58.043493 22326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:58.043531 22327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:56:58.044714 22321 server_base.cc:1034] running on GCE node
I20260929 15:56:58.045876 22321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:56:58.047619 22321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:56:58.048877 22321 hybrid_clock.cc:648] HybridClock initialized: now 1790697418048825 us; error 44 us; skew 500 ppm
I20260929 15:56:58.049317 22321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:56:58 dist-test-slave-r0ck krb5kdc[22308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:56:58.065253 22321 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:56:58.067189 22321 webserver.cc:460] Webserver started at http://127.0.97.62:45489/ using document root <none> and password file <none>
I20260929 15:56:58.067960 22321 fs_manager.cc:359] Metadata directory not provided
I20260929 15:56:58.068109 22321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:56:58.068442 22321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:56:58.071599 22321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "ac9e008206ca4898879905615576172b"
format_stamp: "Formatted at 2026-09-29 15:56:58 on dist-test-slave-r0ck"
server_key: "ad15b272fc835a7b2e86228713e3e346"
server_key_iv: "a1a7ef5e189af0c003743ae250b585ec"
server_key_version: "encryptionkey@0"
I20260929 15:56:58.072379 22321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "ac9e008206ca4898879905615576172b"
format_stamp: "Formatted at 2026-09-29 15:56:58 on dist-test-slave-r0ck"
server_key: "ad15b272fc835a7b2e86228713e3e346"
server_key_iv: "a1a7ef5e189af0c003743ae250b585ec"
server_key_version: "encryptionkey@0"
I20260929 15:56:58.078874 22321 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260929 15:56:58.083498 22336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:56:58.084928 22321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 15:56:58.085112 22321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "ac9e008206ca4898879905615576172b"
format_stamp: "Formatted at 2026-09-29 15:56:58 on dist-test-slave-r0ck"
server_key: "ad15b272fc835a7b2e86228713e3e346"
server_key_iv: "a1a7ef5e189af0c003743ae250b585ec"
server_key_version: "encryptionkey@0"
I20260929 15:56:58.085323 22321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:56:58.091398 22321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:56:58.095631 22321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:56:58.095932 22321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:56:58.115700 22321 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38291
I20260929 15:56:58.115833 22387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38291 every 8 connection(s)
I20260929 15:56:58.117074 22321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:56:58.120971 22388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:56:58.124547 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 22321
I20260929 15:56:58.124763 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:56:58.125216 388 external_mini_cluster.cc:1428] Setting key 873f9858d6a9705104ac08ad39c9c96c
I20260929 15:56:58.132659 22388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac9e008206ca4898879905615576172b: Bootstrap starting.
I20260929 15:56:58.135036 22391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:58.127559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46019 (local address 127.0.97.62:38291)
0929 15:56:58.128654 (+ 1095us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:58.128675 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:58.128979 (+ 304us) server_negotiation.cc:407] Connection header received
0929 15:56:58.131564 (+ 2585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:58.131590 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:58.131946 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:58.132384 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:58.133488 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:58.133494 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:58.133506 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:58.133519 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:58.133570 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:58.133588 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:58.133820 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:58.133946 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 15:56:58.134143 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":773,"thread_start_us":479,"threads_started":1}
I20260929 15:56:58.135926 22388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac9e008206ca4898879905615576172b: Neither blocks nor log segments found. Creating new log.
I20260929 15:56:58.136966 22388 log.cc:824] T 00000000000000000000000000000000 P ac9e008206ca4898879905615576172b: Log is configured to *not* fsync() on all Append() calls
I20260929 15:56:58.138657 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:56:58.140336 22388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac9e008206ca4898879905615576172b: No bootstrap required, opened a new log
I20260929 15:56:58.148137 22391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:56:58.144694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33707 (local address 127.0.97.62:38291)
0929 15:56:58.144929 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 15:56:58.144936 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:56:58.145659 (+ 723us) server_negotiation.cc:407] Connection header received
0929 15:56:58.145853 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:56:58.145857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:56:58.145908 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:56:58.146003 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:56:58.147224 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:56:58.147230 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:56:58.147234 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:56:58.147254 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:56:58.147317 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:56:58.147356 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:56:58.147523 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 15:56:58.147692 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 15:56:58.147757 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 15:56:58.151319 22388 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac9e008206ca4898879905615576172b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac9e008206ca4898879905615576172b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38291 } }
I20260929 15:56:58.151791 22388 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac9e008206ca4898879905615576172b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:56:58.151918 22388 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac9e008206ca4898879905615576172b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9e008206ca4898879905615576172b, State: Initialized, Role: FOLLOWER
I20260929 15:56:58.152700 22388 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac9e008206ca4898879905615576172b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac9e008206ca4898879905615576172b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38291 } }
I20260929 15:56:58.152987 22388 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac9e008206ca4898879905615576172b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:56:58.153149 22388 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac9e008206ca4898879905615576172b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:56:58.153334 22388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac9e008206ca4898879905615576172b [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:56:58.155764 22388 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac9e008206ca4898879905615576172b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac9e008206ca4898879905615576172b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38291 } }
I20260929 15:56:58.156334 22388 leader_election.cc:302] T 00000000000000000000000000000000 P ac9e008206ca4898879905615576172b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac9e008206ca4898879905615576172b; no voters:
I20260929 15:56:58.157076 22388 leader_election.cc:288] T 00000000000000000000000000000000 P ac9e008206ca4898879905615576172b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:56:58.157193 22398 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac9e008206ca4898879905615576172b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:56:58.158115 22398 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac9e008206ca4898879905615576172b [term 1 LEADER]: Becoming Leader. State: Replica: ac9e008206ca4898879905615576172b, State: Running, Role: LEADER
I20260929 15:56:58.158753 22398 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac9e008206ca4898879905615576172b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac9e008206ca4898879905615576172b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38291 } }
I20260929 15:56:58.159602 22388 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac9e008206ca4898879905615576172b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:56:58.162168 22400 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac9e008206ca4898879905615576172b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac9e008206ca4898879905615576172b. Latest consensus state: current_term: 1 leader_uuid: "ac9e008206ca4898879905615576172b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac9e008206ca4898879905615576172b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38291 } } }
I20260929 15:56:58.162680 22400 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac9e008206ca4898879905615576172b [sys.catalog]: This master's current role is: LEADER
I20260929 15:56:58.163344 22399 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac9e008206ca4898879905615576172b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac9e008206ca4898879905615576172b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac9e008206ca4898879905615576172b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38291 } } }
I20260929 15:56:58.164222 22399 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac9e008206ca4898879905615576172b [sys.catalog]: This master's current role is: LEADER
I20260929 15:56:58.170104 22406 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:56:58.175082 22406 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:56:58.183236 22406 catalog_manager.cc:1347] Generated new cluster ID: 26b276c748fc49f0bdc90b0ab60a1124
I20260929 15:56:58.183445 22406 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:56:58.210145 22406 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:56:58.210815 22406 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:56:58.222421 22406 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac9e008206ca4898879905615576172b: Generated new TSK 0
I20260929 15:56:58.223587 22406 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:56:58.236817 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 22321
W20260929 15:56:58.238610 22383 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46019
W20260929 15:56:58.239542 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 22321: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:56:58.239732 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 22321
2026-09-29T15:56:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1979 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:56:58 dist-test-slave-r0ck krb5kdc[22419](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:56:58 dist-test-slave-r0ck krb5kdc[22419](info): set up 2 sockets
Sep 29 15:56:58 dist-test-slave-r0ck krb5kdc[22419](info): commencing operation
krb5kdc: starting...
W20260929 15:56:59.366357 388 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:56:59 dist-test-slave-r0ck krb5kdc[22419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:56:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:56:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:56:59.628978 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37813
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:56:59.955615 22432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:56:59.955945 22432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:56:59.956118 22432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:56:59.963162 22432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:56:59.963310 22432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:56:59.963414 22432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:56:59.963523 22432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:56:59.963608 22432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:56:59.973188 22432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:56:59.974501 22432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:56:59.975769 22432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:56:59.987381 22438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:59.987417 22437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:56:59.987381 22440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:56:59.988173 22432 server_base.cc:1034] running on GCE node
I20260929 15:56:59.988875 22432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:56:59.990242 22432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:56:59.991515 22432 hybrid_clock.cc:648] HybridClock initialized: now 1790697419991475 us; error 32 us; skew 500 ppm
I20260929 15:56:59.991894 22432 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:56:59 dist-test-slave-r0ck krb5kdc[22419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:56:59.995069 22432 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:56:59.996522 22432 webserver.cc:460] Webserver started at http://127.0.97.62:33713/ using document root <none> and password file <none>
I20260929 15:56:59.997169 22432 fs_manager.cc:359] Metadata directory not provided
I20260929 15:56:59.997291 22432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:56:59.997640 22432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:00.000574 22432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "c4f174a4089f4f3ca59a9f5a10313324"
format_stamp: "Formatted at 2026-09-29 15:56:59 on dist-test-slave-r0ck"
server_key: "48564bd2e03a23058f00841b1b09d0b4"
server_key_iv: "585542ee5557fbf2c1294f24ee9e5e76"
server_key_version: "encryptionkey@0"
I20260929 15:57:00.001329 22432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "c4f174a4089f4f3ca59a9f5a10313324"
format_stamp: "Formatted at 2026-09-29 15:56:59 on dist-test-slave-r0ck"
server_key: "48564bd2e03a23058f00841b1b09d0b4"
server_key_iv: "585542ee5557fbf2c1294f24ee9e5e76"
server_key_version: "encryptionkey@0"
I20260929 15:57:00.008050 22432 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260929 15:57:00.017059 22447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:00.019227 22432 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.004s sys 0.002s
I20260929 15:57:00.019662 22432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "c4f174a4089f4f3ca59a9f5a10313324"
format_stamp: "Formatted at 2026-09-29 15:56:59 on dist-test-slave-r0ck"
server_key: "48564bd2e03a23058f00841b1b09d0b4"
server_key_iv: "585542ee5557fbf2c1294f24ee9e5e76"
server_key_version: "encryptionkey@0"
I20260929 15:57:00.019963 22432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:00.037133 22432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:00.041460 22432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:00.041761 22432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:00.061949 22432 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37813
I20260929 15:57:00.061995 22498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37813 every 8 connection(s)
I20260929 15:57:00.063282 22432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:57:00.064584 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 22432
I20260929 15:57:00.064929 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:57:00.065907 388 external_mini_cluster.cc:1428] Setting key 627c61f8ca10092fa52aae313123fa9e
I20260929 15:57:00.069165 22499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 15:57:00 dist-test-slave-r0ck krb5kdc[22419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:57:00.083428 22499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4f174a4089f4f3ca59a9f5a10313324: Bootstrap starting.
I20260929 15:57:00.087175 22499 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4f174a4089f4f3ca59a9f5a10313324: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:00.088222 22499 log.cc:824] T 00000000000000000000000000000000 P c4f174a4089f4f3ca59a9f5a10313324: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:00.090965 22502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:00.067999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43565 (local address 127.0.97.62:37813)
0929 15:57:00.069293 (+ 1294us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:00.069315 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:00.069413 (+ 98us) server_negotiation.cc:407] Connection header received
0929 15:57:00.071053 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:00.071073 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:00.071435 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:00.071777 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:00.073611 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:00.075627 (+ 2016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:00.077665 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:00.078419 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:00.078575 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:00.083434 (+ 4859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:00.083451 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:00.083468 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:00.083510 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:00.086811 (+ 3301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:00.087592 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:00.087599 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:00.087604 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:00.087769 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:00.088181 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:00.088184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:00.088187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:00.088755 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:00.089082 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:00.089578 (+ 496us) server_negotiation.cc:299] Negotiation successful
0929 15:57:00.089977 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":984,"thread_start_us":603,"threads_started":1}
I20260929 15:57:00.091392 22499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4f174a4089f4f3ca59a9f5a10313324: No bootstrap required, opened a new log
I20260929 15:57:00.095036 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:57:00.100641 22499 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4f174a4089f4f3ca59a9f5a10313324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4f174a4089f4f3ca59a9f5a10313324" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37813 } }
I20260929 15:57:00.101042 22499 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4f174a4089f4f3ca59a9f5a10313324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:00.101152 22499 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4f174a4089f4f3ca59a9f5a10313324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f174a4089f4f3ca59a9f5a10313324, State: Initialized, Role: FOLLOWER
I20260929 15:57:00.101783 22499 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4f174a4089f4f3ca59a9f5a10313324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4f174a4089f4f3ca59a9f5a10313324" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37813 } }
I20260929 15:57:00.102033 22499 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4f174a4089f4f3ca59a9f5a10313324 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:57:00.102219 22499 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4f174a4089f4f3ca59a9f5a10313324 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:57:00.102406 22499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4f174a4089f4f3ca59a9f5a10313324 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:00.104589 22499 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4f174a4089f4f3ca59a9f5a10313324 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4f174a4089f4f3ca59a9f5a10313324" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37813 } }
I20260929 15:57:00.105144 22499 leader_election.cc:302] T 00000000000000000000000000000000 P c4f174a4089f4f3ca59a9f5a10313324 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4f174a4089f4f3ca59a9f5a10313324; no voters:
I20260929 15:57:00.105865 22499 leader_election.cc:288] T 00000000000000000000000000000000 P c4f174a4089f4f3ca59a9f5a10313324 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:57:00.105983 22509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4f174a4089f4f3ca59a9f5a10313324 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:00.107473 22509 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4f174a4089f4f3ca59a9f5a10313324 [term 1 LEADER]: Becoming Leader. State: Replica: c4f174a4089f4f3ca59a9f5a10313324, State: Running, Role: LEADER
I20260929 15:57:00.108163 22509 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4f174a4089f4f3ca59a9f5a10313324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4f174a4089f4f3ca59a9f5a10313324" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37813 } }
I20260929 15:57:00.108434 22499 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4f174a4089f4f3ca59a9f5a10313324 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:57:00.111788 22510 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4f174a4089f4f3ca59a9f5a10313324 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4f174a4089f4f3ca59a9f5a10313324" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4f174a4089f4f3ca59a9f5a10313324" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37813 } } }
I20260929 15:57:00.112353 22510 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4f174a4089f4f3ca59a9f5a10313324 [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:00.111801 22511 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4f174a4089f4f3ca59a9f5a10313324 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4f174a4089f4f3ca59a9f5a10313324. Latest consensus state: current_term: 1 leader_uuid: "c4f174a4089f4f3ca59a9f5a10313324" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4f174a4089f4f3ca59a9f5a10313324" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37813 } } }
I20260929 15:57:00.112725 22511 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4f174a4089f4f3ca59a9f5a10313324 [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:00.117079 22502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:00.099279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59139 (local address 127.0.97.62:37813)
0929 15:57:00.099472 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:00.099481 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:00.099845 (+ 364us) server_negotiation.cc:407] Connection header received
0929 15:57:00.100044 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:00.100053 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:00.100111 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:00.100211 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:00.101805 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:00.102751 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:00.104005 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:00.104445 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:00.104585 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:00.107315 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:00.107324 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:00.107331 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:00.107381 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:00.112374 (+ 4993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:00.113275 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:00.113284 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:00.113290 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:00.113535 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:00.114124 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:00.114131 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:00.114136 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:00.114617 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:00.114982 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:00.116213 (+ 1231us) server_negotiation.cc:299] Negotiation successful
0929 15:57:00.116521 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 15:57:00.121201 22515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:57:00.125962 22515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:57:00.133949 22515 catalog_manager.cc:1347] Generated new cluster ID: 3e383cc1efe249dbb5b03e191587d3c7
I20260929 15:57:00.134075 22515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:57:00.150055 22515 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:57:00.151371 22515 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:57:00.157792 22515 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4f174a4089f4f3ca59a9f5a10313324: Generated new TSK 0
I20260929 15:57:00.158664 22515 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:57:00.170683 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 22432
I20260929 15:57:00.171753 22494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:00.261770 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 22432
2026-09-29T15:57:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2022 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:57:00 dist-test-slave-r0ck krb5kdc[22531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:57:00 dist-test-slave-r0ck krb5kdc[22531](info): set up 2 sockets
Sep 29 15:57:00 dist-test-slave-r0ck krb5kdc[22531](info): commencing operation
krb5kdc: starting...
W20260929 15:57:01.399474 388 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:57:01 dist-test-slave-r0ck krb5kdc[22531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:57:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:57:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:57:01.669560 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42935
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:02.000504 22544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:02.000900 22544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:02.001097 22544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:02.009424 22544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:57:02.009624 22544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:02.009742 22544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:02.009855 22544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:57:02.009948 22544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:57:02.021653 22544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:57:02.023178 22544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:02.024593 22544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:02.037230 22549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:02.037222 22550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:02.037525 22552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:02.038142 22544 server_base.cc:1034] running on GCE node
I20260929 15:57:02.038983 22544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:02.040576 22544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:02.041790 22544 hybrid_clock.cc:648] HybridClock initialized: now 1790697422041732 us; error 40 us; skew 500 ppm
I20260929 15:57:02.042168 22544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:02 dist-test-slave-r0ck krb5kdc[22531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:02.045713 22544 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:57:02.047936 22544 webserver.cc:460] Webserver started at http://127.0.97.62:36165/ using document root <none> and password file <none>
I20260929 15:57:02.049026 22544 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:02.049222 22544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:02.049607 22544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:02.052676 22544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "f9c3e4f1721648b9b03e52d913c2e0c7"
format_stamp: "Formatted at 2026-09-29 15:57:02 on dist-test-slave-r0ck"
server_key: "42a7ed06107e9ec4966366b41d6ee696"
server_key_iv: "8f8327fff17cdff6181a5aad6c33c698"
server_key_version: "encryptionkey@0"
I20260929 15:57:02.053561 22544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "f9c3e4f1721648b9b03e52d913c2e0c7"
format_stamp: "Formatted at 2026-09-29 15:57:02 on dist-test-slave-r0ck"
server_key: "42a7ed06107e9ec4966366b41d6ee696"
server_key_iv: "8f8327fff17cdff6181a5aad6c33c698"
server_key_version: "encryptionkey@0"
I20260929 15:57:02.059527 22544 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 15:57:02.063221 22559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:02.064849 22544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 15:57:02.065049 22544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "f9c3e4f1721648b9b03e52d913c2e0c7"
format_stamp: "Formatted at 2026-09-29 15:57:02 on dist-test-slave-r0ck"
server_key: "42a7ed06107e9ec4966366b41d6ee696"
server_key_iv: "8f8327fff17cdff6181a5aad6c33c698"
server_key_version: "encryptionkey@0"
I20260929 15:57:02.065253 22544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:02.085727 22544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:02.090462 22544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:02.090785 22544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:02.110980 22544 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42935
I20260929 15:57:02.111001 22610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42935 every 8 connection(s)
I20260929 15:57:02.112437 22544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:57:02.114966 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 22544
I20260929 15:57:02.115147 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:57:02.115715 388 external_mini_cluster.cc:1428] Setting key 688dc72c3a54b4eebc494c9e3744ccbc
I20260929 15:57:02.116781 22611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:02.128868 22614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:02.117609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47403 (local address 127.0.97.62:42935)
0929 15:57:02.118435 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:02.118446 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:02.118485 (+ 39us) server_negotiation.cc:407] Connection header received
0929 15:57:02.120190 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:02.120212 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:02.120600 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:02.121022 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:02.121596 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:02.122939 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:02.124211 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:02.124875 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:02.125063 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:02.126545 (+ 1482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:02.126552 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:02.126566 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:57:02.127139 (+ 573us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:02.127229 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:57:02.127257 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:02.127386 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:02.127651 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 15:57:02.127918 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":395,"threads_started":1}
I20260929 15:57:02.129945 22611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9c3e4f1721648b9b03e52d913c2e0c7: Bootstrap starting.
I20260929 15:57:02.132987 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:57:02.133983 22611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9c3e4f1721648b9b03e52d913c2e0c7: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:02.135696 22611 log.cc:824] T 00000000000000000000000000000000 P f9c3e4f1721648b9b03e52d913c2e0c7: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:02.138814 22611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9c3e4f1721648b9b03e52d913c2e0c7: No bootstrap required, opened a new log
I20260929 15:57:02.146018 22614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:02.139323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39965 (local address 127.0.97.62:42935)
0929 15:57:02.139515 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:02.139520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:02.139905 (+ 385us) server_negotiation.cc:407] Connection header received
0929 15:57:02.140052 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:02.140060 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:02.140141 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:02.140267 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:02.140767 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:02.141925 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:02.143286 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:02.144005 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:02.144141 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:02.145170 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:02.145176 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:02.145181 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:57:02.145202 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:02.145258 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:57:02.145278 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:02.145401 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:02.145575 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 15:57:02.145672 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 15:57:02.148689 22611 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9c3e4f1721648b9b03e52d913c2e0c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c3e4f1721648b9b03e52d913c2e0c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42935 } }
I20260929 15:57:02.149026 22611 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9c3e4f1721648b9b03e52d913c2e0c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:02.149096 22611 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9c3e4f1721648b9b03e52d913c2e0c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9c3e4f1721648b9b03e52d913c2e0c7, State: Initialized, Role: FOLLOWER
I20260929 15:57:02.149634 22611 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9c3e4f1721648b9b03e52d913c2e0c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c3e4f1721648b9b03e52d913c2e0c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42935 } }
I20260929 15:57:02.149837 22611 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9c3e4f1721648b9b03e52d913c2e0c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:57:02.149924 22611 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9c3e4f1721648b9b03e52d913c2e0c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:57:02.150049 22611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9c3e4f1721648b9b03e52d913c2e0c7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:02.153259 22611 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9c3e4f1721648b9b03e52d913c2e0c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c3e4f1721648b9b03e52d913c2e0c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42935 } }
I20260929 15:57:02.154107 22611 leader_election.cc:302] T 00000000000000000000000000000000 P f9c3e4f1721648b9b03e52d913c2e0c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9c3e4f1721648b9b03e52d913c2e0c7; no voters:
I20260929 15:57:02.154910 22611 leader_election.cc:288] T 00000000000000000000000000000000 P f9c3e4f1721648b9b03e52d913c2e0c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:57:02.155153 22621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9c3e4f1721648b9b03e52d913c2e0c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:02.156179 22621 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9c3e4f1721648b9b03e52d913c2e0c7 [term 1 LEADER]: Becoming Leader. State: Replica: f9c3e4f1721648b9b03e52d913c2e0c7, State: Running, Role: LEADER
I20260929 15:57:02.156894 22621 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9c3e4f1721648b9b03e52d913c2e0c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c3e4f1721648b9b03e52d913c2e0c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42935 } }
I20260929 15:57:02.157425 22611 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9c3e4f1721648b9b03e52d913c2e0c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:57:02.159601 22622 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9c3e4f1721648b9b03e52d913c2e0c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9c3e4f1721648b9b03e52d913c2e0c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c3e4f1721648b9b03e52d913c2e0c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42935 } } }
I20260929 15:57:02.159973 22622 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9c3e4f1721648b9b03e52d913c2e0c7 [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:02.160434 22623 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9c3e4f1721648b9b03e52d913c2e0c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9c3e4f1721648b9b03e52d913c2e0c7. Latest consensus state: current_term: 1 leader_uuid: "f9c3e4f1721648b9b03e52d913c2e0c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c3e4f1721648b9b03e52d913c2e0c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42935 } } }
I20260929 15:57:02.160746 22623 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9c3e4f1721648b9b03e52d913c2e0c7 [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:02.161074 22627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:57:02.167510 22627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:57:02.176270 22627 catalog_manager.cc:1347] Generated new cluster ID: a92d38b53e1142ee88f1e9f8da246169
I20260929 15:57:02.176386 22627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:57:02.198180 22627 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:57:02.199460 22627 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:57:02.206198 22627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9c3e4f1721648b9b03e52d913c2e0c7: Generated new TSK 0
I20260929 15:57:02.207059 22627 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:57:02.212773 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 22544
W20260929 15:57:02.213971 22606 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47403
W20260929 15:57:02.214803 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 22544: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:57:02.214969 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 22544
2026-09-29T15:57:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1960 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:57:02 dist-test-slave-r0ck krb5kdc[22642](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:57:02 dist-test-slave-r0ck krb5kdc[22642](info): set up 2 sockets
Sep 29 15:57:02 dist-test-slave-r0ck krb5kdc[22642](info): commencing operation
krb5kdc: starting...
W20260929 15:57:03.337735 388 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:57:03 dist-test-slave-r0ck krb5kdc[22642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:57:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:57:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:57:03.593492 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37175
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:03.936964 22655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:03.937237 22655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:03.937412 22655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:03.944545 22655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:57:03.944676 22655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:03.944778 22655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:03.944890 22655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:57:03.944971 22655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:57:03.954629 22655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:57:03.955926 22655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:03.957187 22655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:03.968430 22660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:03.968459 22663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:03.968490 22661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:03.969226 22655 server_base.cc:1034] running on GCE node
I20260929 15:57:03.970046 22655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:03.971334 22655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:03.972546 22655 hybrid_clock.cc:648] HybridClock initialized: now 1790697423972501 us; error 34 us; skew 500 ppm
I20260929 15:57:03.972882 22655 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:03 dist-test-slave-r0ck krb5kdc[22642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:03.979521 22655 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:57:03.981040 22655 webserver.cc:460] Webserver started at http://127.0.97.62:42721/ using document root <none> and password file <none>
I20260929 15:57:03.981778 22655 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:03.981881 22655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:03.982178 22655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:03.985041 22655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "96cf970d128741fc9e9b2333fc754df3"
format_stamp: "Formatted at 2026-09-29 15:57:03 on dist-test-slave-r0ck"
server_key: "a86148bbb9f02e1f2bd6c390aa72b93b"
server_key_iv: "c2f39c508c44941d4488a843a5e01bcf"
server_key_version: "encryptionkey@0"
I20260929 15:57:03.985831 22655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "96cf970d128741fc9e9b2333fc754df3"
format_stamp: "Formatted at 2026-09-29 15:57:03 on dist-test-slave-r0ck"
server_key: "a86148bbb9f02e1f2bd6c390aa72b93b"
server_key_iv: "c2f39c508c44941d4488a843a5e01bcf"
server_key_version: "encryptionkey@0"
I20260929 15:57:03.990782 22655 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 15:57:03.994220 22670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:03.995494 22655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 15:57:03.995657 22655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "96cf970d128741fc9e9b2333fc754df3"
format_stamp: "Formatted at 2026-09-29 15:57:03 on dist-test-slave-r0ck"
server_key: "a86148bbb9f02e1f2bd6c390aa72b93b"
server_key_iv: "c2f39c508c44941d4488a843a5e01bcf"
server_key_version: "encryptionkey@0"
I20260929 15:57:03.995851 22655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:04.001528 22655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:04.005950 22655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:04.006256 22655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:04.026600 22655 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37175
I20260929 15:57:04.026643 22721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37175 every 8 connection(s)
I20260929 15:57:04.028131 22655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:57:04.030797 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 22655
I20260929 15:57:04.030972 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:57:04.031517 388 external_mini_cluster.cc:1428] Setting key 824b629193da043501fce9ba80589311
I20260929 15:57:04.032927 22722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:04.039067 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:04.033500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40025 (local address 127.0.97.62:37175)
0929 15:57:04.034280 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:04.034291 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:04.034330 (+ 39us) server_negotiation.cc:407] Connection header received
0929 15:57:04.035824 (+ 1494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:04.035844 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:04.036156 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:04.036491 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:04.037549 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:04.037554 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:04.037565 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:57:04.037580 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:04.037661 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:57:04.037680 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:04.037844 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:04.037943 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 15:57:04.038142 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":368,"threads_started":1}
I20260929 15:57:04.042464 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:57:04.045069 22722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96cf970d128741fc9e9b2333fc754df3: Bootstrap starting.
I20260929 15:57:04.048295 22722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96cf970d128741fc9e9b2333fc754df3: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:04.048965 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:04.046454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50727 (local address 127.0.97.62:37175)
0929 15:57:04.046632 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:04.046638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:04.047110 (+ 472us) server_negotiation.cc:407] Connection header received
0929 15:57:04.047226 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:04.047230 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:04.047296 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:04.047399 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:04.048326 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:04.048331 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:04.048335 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:57:04.048354 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:04.048406 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:57:04.048426 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:04.048541 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:04.048624 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 15:57:04.048685 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 15:57:04.049546 22722 log.cc:824] T 00000000000000000000000000000000 P 96cf970d128741fc9e9b2333fc754df3: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:04.052030 22722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96cf970d128741fc9e9b2333fc754df3: No bootstrap required, opened a new log
I20260929 15:57:04.059134 22722 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96cf970d128741fc9e9b2333fc754df3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96cf970d128741fc9e9b2333fc754df3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37175 } }
I20260929 15:57:04.059506 22722 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96cf970d128741fc9e9b2333fc754df3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:04.059612 22722 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96cf970d128741fc9e9b2333fc754df3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96cf970d128741fc9e9b2333fc754df3, State: Initialized, Role: FOLLOWER
I20260929 15:57:04.060249 22722 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96cf970d128741fc9e9b2333fc754df3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96cf970d128741fc9e9b2333fc754df3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37175 } }
I20260929 15:57:04.060523 22722 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96cf970d128741fc9e9b2333fc754df3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:57:04.060695 22722 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96cf970d128741fc9e9b2333fc754df3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:57:04.060884 22722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96cf970d128741fc9e9b2333fc754df3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:04.063059 22722 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96cf970d128741fc9e9b2333fc754df3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96cf970d128741fc9e9b2333fc754df3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37175 } }
I20260929 15:57:04.063478 22722 leader_election.cc:302] T 00000000000000000000000000000000 P 96cf970d128741fc9e9b2333fc754df3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96cf970d128741fc9e9b2333fc754df3; no voters:
I20260929 15:57:04.063967 22722 leader_election.cc:288] T 00000000000000000000000000000000 P 96cf970d128741fc9e9b2333fc754df3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:57:04.064105 22732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96cf970d128741fc9e9b2333fc754df3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:04.064795 22732 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96cf970d128741fc9e9b2333fc754df3 [term 1 LEADER]: Becoming Leader. State: Replica: 96cf970d128741fc9e9b2333fc754df3, State: Running, Role: LEADER
I20260929 15:57:04.065326 22732 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96cf970d128741fc9e9b2333fc754df3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96cf970d128741fc9e9b2333fc754df3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37175 } }
I20260929 15:57:04.066067 22722 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96cf970d128741fc9e9b2333fc754df3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:57:04.068264 22734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96cf970d128741fc9e9b2333fc754df3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96cf970d128741fc9e9b2333fc754df3. Latest consensus state: current_term: 1 leader_uuid: "96cf970d128741fc9e9b2333fc754df3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96cf970d128741fc9e9b2333fc754df3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37175 } } }
I20260929 15:57:04.068666 22734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96cf970d128741fc9e9b2333fc754df3 [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:04.069525 22733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96cf970d128741fc9e9b2333fc754df3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96cf970d128741fc9e9b2333fc754df3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96cf970d128741fc9e9b2333fc754df3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37175 } } }
I20260929 15:57:04.069854 22733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96cf970d128741fc9e9b2333fc754df3 [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:04.071619 22740 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:57:04.076828 22740 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:57:04.085459 22740 catalog_manager.cc:1347] Generated new cluster ID: 65f97ee752f241c18bed7e5c39792e7f
I20260929 15:57:04.085562 22740 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:57:04.106727 22740 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:57:04.107124 22740 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:57:04.114513 22740 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96cf970d128741fc9e9b2333fc754df3: Generated new TSK 0
I20260929 15:57:04.115428 22740 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:57:04.124888 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 22655
W20260929 15:57:04.126029 22717 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40025
W20260929 15:57:04.126739 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 22655: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:57:04.126890 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 22655
2026-09-29T15:57:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1906 ms)
[----------] 18 tests from AuthTokenIssuingTest (35031 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:57:04 dist-test-slave-r0ck krb5kdc[22753](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:57:04 dist-test-slave-r0ck krb5kdc[22753](info): set up 2 sockets
Sep 29 15:57:04 dist-test-slave-r0ck krb5kdc[22753](info): commencing operation
krb5kdc: starting...
W20260929 15:57:05.258189 388 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.005s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:57:05 dist-test-slave-r0ck krb5kdc[22753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:57:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:57:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:57:05.529453 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33163
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33163,127.0.97.61:44099,127.0.97.60:33763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:05.871469 22766 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 15:57:05.871837 22766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:05.871943 22766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:05.872076 22766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:05.880496 22766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:57:05.880607 22766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:05.880671 22766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:05.880788 22766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:57:05.880893 22766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:57:05.891657 22766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33163,127.0.97.61:44099,127.0.97.60:33763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:57:05.893200 22766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:05.894577 22766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:05.909883 22771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:05.909803 22774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:05.910002 22772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:05.911911 22766 server_base.cc:1034] running on GCE node
I20260929 15:57:05.912793 22766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:05.914335 22766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:05.915601 22766 hybrid_clock.cc:648] HybridClock initialized: now 1790697425915526 us; error 61 us; skew 500 ppm
I20260929 15:57:05.915930 22766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:05 dist-test-slave-r0ck krb5kdc[22753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:05.919170 22766 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:57:05.920833 22766 webserver.cc:460] Webserver started at http://127.0.97.62:37567/ using document root <none> and password file <none>
I20260929 15:57:05.921602 22766 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:05.921702 22766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:05.922047 22766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:05.925097 22766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "bcef83f3630a40c5a0bdf92c49e56fca"
format_stamp: "Formatted at 2026-09-29 15:57:05 on dist-test-slave-r0ck"
server_key: "ff961b94ef6712598b0569e76e860100"
server_key_iv: "6cc1f444656bdb2804bde2303c109f39"
server_key_version: "encryptionkey@0"
I20260929 15:57:05.926000 22766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "bcef83f3630a40c5a0bdf92c49e56fca"
format_stamp: "Formatted at 2026-09-29 15:57:05 on dist-test-slave-r0ck"
server_key: "ff961b94ef6712598b0569e76e860100"
server_key_iv: "6cc1f444656bdb2804bde2303c109f39"
server_key_version: "encryptionkey@0"
I20260929 15:57:05.931697 22766 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 15:57:05.935698 22781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:05.937325 22766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 15:57:05.937583 22766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "bcef83f3630a40c5a0bdf92c49e56fca"
format_stamp: "Formatted at 2026-09-29 15:57:05 on dist-test-slave-r0ck"
server_key: "ff961b94ef6712598b0569e76e860100"
server_key_iv: "6cc1f444656bdb2804bde2303c109f39"
server_key_version: "encryptionkey@0"
I20260929 15:57:05.937848 22766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:05.963552 22766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:05.968616 22766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:05.968925 22766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:05.989450 22766 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33163
I20260929 15:57:05.989538 22832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33163 every 8 connection(s)
I20260929 15:57:05.990792 22766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:57:05.994503 22833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:05.996529 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 22766
I20260929 15:57:05.996763 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:57:05.997418 388 external_mini_cluster.cc:1428] Setting key d5bc31bec54d3873a12f43cd44ac2b2a
I20260929 15:57:06.001180 22833 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33163 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:57:06 dist-test-slave-r0ck krb5kdc[22753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260929 15:57:06.024845 22835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:06.004447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43924 (local address 127.0.97.62:33163)
0929 15:57:06.005964 (+ 1517us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:06.005991 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:06.006509 (+ 518us) server_negotiation.cc:407] Connection header received
0929 15:57:06.008655 (+ 2146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:06.008660 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:06.008873 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:06.009072 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:06.010555 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:06.011786 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:06.013131 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:06.013719 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:06.013932 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:06.017260 (+ 3328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:06.017269 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:06.017281 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:06.017311 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:06.020354 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:06.021129 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:06.021135 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:06.021138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:06.021283 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:06.021716 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:06.021721 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:06.021725 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:06.022242 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:06.022448 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:06.022997 (+ 549us) server_negotiation.cc:299] Negotiation successful
0929 15:57:06.023403 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":1096,"thread_start_us":739,"threads_started":1}
I20260929 15:57:06.024868 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:06.004448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33163 (local address 127.0.0.1:43924)
0929 15:57:06.006326 (+ 1878us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:06.006383 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:06.008269 (+ 1886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:06.009103 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:06.009109 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:06.009841 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:06.010297 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:06.010320 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:06.011947 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:06.011957 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:06.012986 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:06.012998 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:06.013985 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:06.013991 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:06.014166 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:06.014869 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:06.014891 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:06.017028 (+ 2137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:06.020505 (+ 3477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:06.020519 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:06.020532 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:06.020971 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:06.021421 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:06.021426 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:06.021429 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:06.021612 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:06.022388 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:06.022395 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:06.022618 (+ 223us) client_negotiation.cc:769] Sending connection context
0929 15:57:06.022864 (+ 246us) client_negotiation.cc:240] Negotiation successful
0929 15:57:06.023517 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1657,"mutex_wait_us":181,"thread_start_us":1213,"threads_started":1}
I20260929 15:57:06.028024 22833 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44099 } has no permanent_uuid. Determining permanent_uuid...
I20260929 15:57:06.029410 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:06.028748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44099 (local address 127.0.0.1:43082)
0929 15:57:06.029027 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:06.029163 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
W20260929 15:57:06.029724 22784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44099: connect: Connection refused (error 111)
I20260929 15:57:06.031739 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:06.030958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44099 (local address 127.0.0.1:43088)
0929 15:57:06.031277 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:06.031522 (+ 245us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260929 15:57:06.032085 22833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44099: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44099: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260929 15:57:06.058640 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:44099
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33163,127.0.97.61:44099,127.0.97.60:33763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:57:06.081483 22833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44099 } attempt: 1
I20260929 15:57:06.082893 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:06.082281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44099 (local address 127.0.0.1:43104)
0929 15:57:06.082546 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:06.082684 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 15:57:06.084002 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:06.083628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44099 (local address 127.0.0.1:43108)
0929 15:57:06.083774 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:06.083854 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 15:57:06.084230 22833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44099: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44099: connect: Connection refused (error 111)
I20260929 15:57:06.152521 22833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44099 } attempt: 2
I20260929 15:57:06.153864 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:06.153263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44099 (local address 127.0.0.1:43112)
0929 15:57:06.153560 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:06.153694 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260929 15:57:06.155045 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:06.154564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44099 (local address 127.0.0.1:43126)
0929 15:57:06.154755 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:06.154873 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 15:57:06.155438 22833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44099: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44099: connect: Connection refused (error 111)
I20260929 15:57:06.246776 22833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44099 } attempt: 3
I20260929 15:57:06.248087 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:06.247436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44099 (local address 127.0.0.1:43134)
0929 15:57:06.247712 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:06.247845 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260929 15:57:06.250069 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:06.249565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44099 (local address 127.0.0.1:43136)
0929 15:57:06.249754 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:06.249859 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 15:57:06.250339 22833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44099: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44099: connect: Connection refused (error 111)
W20260929 15:57:06.386268 22840 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 15:57:06.386636 22840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:06.386771 22840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:06.386893 22840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 15:57:06.393697 22833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44099 } attempt: 4
W20260929 15:57:06.394021 22840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:57:06.394196 22840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:06.394302 22840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:06.394382 22840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:57:06.394474 22840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:57:06.395386 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:06.394768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44099 (local address 127.0.0.1:43150)
0929 15:57:06.395115 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:06.395225 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
I20260929 15:57:06.396450 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:06.396087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44099 (local address 127.0.0.1:43158)
0929 15:57:06.396233 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:06.396312 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 15:57:06.396799 22833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44099: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44099: connect: Connection refused (error 111)
I20260929 15:57:06.405330 22840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33163,127.0.97.61:44099,127.0.97.60:33763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:44099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:57:06.406747 22840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:06.408416 22840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:06.423486 22845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:06.423805 22846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:06.424144 22848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:06.424635 22840 server_base.cc:1034] running on GCE node
I20260929 15:57:06.425547 22840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:06.426976 22840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:06.428248 22840 hybrid_clock.cc:648] HybridClock initialized: now 1790697426428189 us; error 39 us; skew 500 ppm
I20260929 15:57:06.428776 22840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:06 dist-test-slave-r0ck krb5kdc[22753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:06.432302 22840 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260929 15:57:06.434537 22840 webserver.cc:460] Webserver started at http://127.0.97.61:39229/ using document root <none> and password file <none>
I20260929 15:57:06.435578 22840 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:06.435705 22840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:06.435981 22840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:06.439080 22840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-1/data/instance:
uuid: "626d03afb1f348abb00119bb77b7261d"
format_stamp: "Formatted at 2026-09-29 15:57:06 on dist-test-slave-r0ck"
server_key: "c95811ddf645f02027a275594f4812ca"
server_key_iv: "070c23fdc3c8ce56ce976654fa3f61f8"
server_key_version: "encryptionkey@0"
I20260929 15:57:06.439868 22840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-1/wal/instance:
uuid: "626d03afb1f348abb00119bb77b7261d"
format_stamp: "Formatted at 2026-09-29 15:57:06 on dist-test-slave-r0ck"
server_key: "c95811ddf645f02027a275594f4812ca"
server_key_iv: "070c23fdc3c8ce56ce976654fa3f61f8"
server_key_version: "encryptionkey@0"
I20260929 15:57:06.445693 22840 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260929 15:57:06.449774 22855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:06.451611 22840 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 15:57:06.451784 22840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-1/wal
uuid: "626d03afb1f348abb00119bb77b7261d"
format_stamp: "Formatted at 2026-09-29 15:57:06 on dist-test-slave-r0ck"
server_key: "c95811ddf645f02027a275594f4812ca"
server_key_iv: "070c23fdc3c8ce56ce976654fa3f61f8"
server_key_version: "encryptionkey@0"
I20260929 15:57:06.451977 22840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:06.478350 22840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:06.483013 22840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:06.483250 22840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:06.503131 22840 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:44099
I20260929 15:57:06.503168 22906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:44099 every 8 connection(s)
I20260929 15:57:06.504535 22840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-1/data/info.pb
I20260929 15:57:06.508258 22907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:06.514309 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 22840
I20260929 15:57:06.514497 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-1/wal/instance
I20260929 15:57:06.514928 388 external_mini_cluster.cc:1428] Setting key e3723bf7dc6fda0a0d885f73656238e0
I20260929 15:57:06.514806 22907 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33163 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:57:06 dist-test-slave-r0ck krb5kdc[22753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:57:06.535115 22835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:06.517441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43930 (local address 127.0.97.62:33163)
0929 15:57:06.517674 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:06.517683 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:06.518850 (+ 1167us) server_negotiation.cc:407] Connection header received
0929 15:57:06.520665 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:06.520674 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:06.520782 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:06.520923 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:06.522459 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:06.523154 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:06.524635 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:06.525042 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:06.525187 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:06.528960 (+ 3773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:06.528965 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:06.528968 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:06.529000 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:06.531724 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:06.532666 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:06.532670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:06.532672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:06.532805 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:06.533296 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:06.533300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:06.533302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:06.533651 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:06.533787 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:06.534552 (+ 765us) server_negotiation.cc:299] Negotiation successful
0929 15:57:06.534789 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 15:57:06.536079 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:06.517877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33163 (local address 127.0.0.1:43930)
0929 15:57:06.518669 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:06.518720 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:06.520406 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:06.521086 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:06.521093 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:06.521719 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:06.522279 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:06.522296 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:06.523323 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:06.523328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:06.524478 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:06.524489 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:06.525284 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:06.525292 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:06.525560 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:06.526253 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:06.526274 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:06.528710 (+ 2436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:06.531867 (+ 3157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:06.531878 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:06.531897 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:06.532535 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:06.532920 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:06.532928 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:06.532934 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:06.533162 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:06.533866 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:06.533872 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:06.534181 (+ 309us) client_negotiation.cc:769] Sending connection context
0929 15:57:06.534444 (+ 263us) client_negotiation.cc:240] Negotiation successful
0929 15:57:06.534951 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":358,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260929 15:57:06.537803 22907 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44099 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Sep 29 15:57:06 dist-test-slave-r0ck krb5kdc[22753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 15:57:06.552403 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:06.538599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44099 (local address 127.0.0.1:43160)
0929 15:57:06.538865 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:06.538891 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:06.539016 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:06.539667 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:06.539671 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:06.540050 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:06.540219 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:06.540228 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:06.541783 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:06.541790 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:06.542727 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:06.542739 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:06.543530 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:06.543535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:06.543768 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:06.544354 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:06.544370 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:06.546446 (+ 2076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:06.549961 (+ 3515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:06.549966 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:06.549971 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:06.550319 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:06.550672 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:06.550678 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:06.550681 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:06.550801 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:06.551380 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:06.551390 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:06.551555 (+ 165us) client_negotiation.cc:769] Sending connection context
0929 15:57:06.551708 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 15:57:06.551985 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":3}
I20260929 15:57:06.552490 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:06.538669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43160 (local address 127.0.97.61:44099)
0929 15:57:06.539230 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:06.539235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:06.539260 (+ 25us) server_negotiation.cc:407] Connection header received
0929 15:57:06.539302 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:06.539307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:06.539469 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:06.539576 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:06.540413 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:06.541644 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:06.542854 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:06.543390 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:06.543521 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:06.546654 (+ 3133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:06.546665 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:06.546679 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:06.546707 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:06.549823 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:06.550445 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:06.550460 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:06.550463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:06.550566 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:06.550910 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:06.550914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:06.550917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:06.551248 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:06.551477 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:06.551842 (+ 365us) server_negotiation.cc:299] Negotiation successful
0929 15:57:06.552100 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":425,"thread_start_us":282,"threads_started":1}
I20260929 15:57:06.555287 22907 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33763 } has no permanent_uuid. Determining permanent_uuid...
I20260929 15:57:06.556563 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:06.555964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33763 (local address 127.0.0.1:38252)
0929 15:57:06.556264 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:06.556405 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
W20260929 15:57:06.556809 22858 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33763: connect: Connection refused (error 111)
I20260929 15:57:06.558441 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:06.558088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33763 (local address 127.0.0.1:38258)
0929 15:57:06.558266 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:06.558339 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 15:57:06.558674 22907 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33763: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33763: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260929 15:57:06.573115 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:33763
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33163,127.0.97.61:44099,127.0.97.60:33763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:57:06.607962 22907 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33763 } attempt: 1
I20260929 15:57:06.609203 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:06.608674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33763 (local address 127.0.0.1:38268)
0929 15:57:06.608905 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:06.609019 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 15:57:06.610278 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:06.609874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33763 (local address 127.0.0.1:38272)
0929 15:57:06.610046 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:06.610142 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 15:57:06.610528 22907 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33763: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33763: connect: Connection refused (error 111)
I20260929 15:57:06.678774 22907 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33763 } attempt: 2
I20260929 15:57:06.680097 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:06.679619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33763 (local address 127.0.0.1:38276)
0929 15:57:06.679858 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:06.679948 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260929 15:57:06.681156 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:06.680824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33763 (local address 127.0.0.1:38290)
0929 15:57:06.680972 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:06.681047 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 15:57:06.681386 22907 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33763: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33763: connect: Connection refused (error 111)
I20260929 15:57:06.696089 22833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44099 } attempt: 5
Sep 29 15:57:06 dist-test-slave-r0ck krb5kdc[22753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 15:57:06.710757 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:06.696861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44099 (local address 127.0.0.1:43170)
0929 15:57:06.697047 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:06.697067 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:06.697192 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:06.697582 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:06.697585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:06.697961 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:06.698130 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:06.698139 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:06.699535 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:06.699539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:06.700383 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:06.700394 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:06.701440 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:06.701444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:06.701606 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:06.702285 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:06.702302 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:06.704765 (+ 2463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:06.708496 (+ 3731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:06.708500 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:06.708503 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:06.708862 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:06.709252 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:06.709255 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:06.709258 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:06.709442 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:06.710049 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:06.710052 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:06.710197 (+ 145us) client_negotiation.cc:769] Sending connection context
0929 15:57:06.710313 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 15:57:06.710510 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 15:57:06.711081 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:06.696965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43170 (local address 127.0.97.61:44099)
0929 15:57:06.697140 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:06.697146 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:06.697169 (+ 23us) server_negotiation.cc:407] Connection header received
0929 15:57:06.697316 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:06.697322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:06.697435 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:06.697548 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:06.698287 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:06.699373 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:06.700523 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:06.701236 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:06.701401 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:06.704965 (+ 3564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:06.704971 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:06.704975 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:06.705013 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:06.708364 (+ 3351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:06.708986 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:06.708992 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:06.708995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:06.709139 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:06.709546 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:06.709551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:06.709554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:06.709934 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:06.710084 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:06.710467 (+ 383us) server_negotiation.cc:299] Negotiation successful
0929 15:57:06.710688 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 15:57:06.712189 22833 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33763 } has no permanent_uuid. Determining permanent_uuid...
I20260929 15:57:06.713282 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:06.712879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33763 (local address 127.0.0.1:38294)
0929 15:57:06.713041 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:06.713145 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 15:57:06.714205 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:06.713893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33763 (local address 127.0.0.1:38302)
0929 15:57:06.714032 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:06.714111 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 15:57:06.714403 22833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33763: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33763: connect: Connection refused (error 111)
I20260929 15:57:06.765647 22833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33763 } attempt: 1
I20260929 15:57:06.766996 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:06.766493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33763 (local address 127.0.0.1:38314)
0929 15:57:06.766722 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:06.766836 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260929 15:57:06.767989 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:06.767684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33763 (local address 127.0.0.1:38330)
0929 15:57:06.767824 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:06.767897 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 15:57:06.768206 22833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33763: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33763: connect: Connection refused (error 111)
I20260929 15:57:06.772604 22907 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33763 } attempt: 3
I20260929 15:57:06.773619 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:06.773131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33763 (local address 127.0.0.1:38342)
0929 15:57:06.773377 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:06.773477 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 15:57:06.774880 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:06.774450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33763 (local address 127.0.0.1:38354)
0929 15:57:06.774680 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:06.774761 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260929 15:57:06.775087 22907 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33763: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33763: connect: Connection refused (error 111)
I20260929 15:57:06.836427 22833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33763 } attempt: 2
I20260929 15:57:06.837596 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:06.837079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33763 (local address 127.0.0.1:38358)
0929 15:57:06.837293 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:06.837437 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 15:57:06.838514 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:06.838178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33763 (local address 127.0.0.1:38370)
0929 15:57:06.838343 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:06.838422 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260929 15:57:06.838774 22833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33763: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33763: connect: Connection refused (error 111)
W20260929 15:57:06.907117 22914 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 15:57:06.907490 22914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:06.907617 22914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:06.907752 22914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:06.915045 22914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:57:06.915179 22914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:06.915308 22914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:06.915395 22914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:57:06.915486 22914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:57:06.918327 22907 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33763 } attempt: 4
I20260929 15:57:06.919431 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:06.918950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33763 (local address 127.0.0.1:38372)
0929 15:57:06.919185 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:06.919264 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 15:57:06.920363 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:06.920062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33763 (local address 127.0.0.1:38386)
0929 15:57:06.920207 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:06.920271 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 15:57:06.920557 22907 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33763: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33763: connect: Connection refused (error 111)
I20260929 15:57:06.925936 22914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33163,127.0.97.61:44099,127.0.97.60:33763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:33763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:57:06.927280 22914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:06.928622 22914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:06.941286 22920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:06.941372 22922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:06.941391 22919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:06.942138 22914 server_base.cc:1034] running on GCE node
I20260929 15:57:06.942893 22914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:06.944249 22914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:06.944968 22833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33763 } attempt: 3
I20260929 15:57:06.945442 22914 hybrid_clock.cc:648] HybridClock initialized: now 1790697426945354 us; error 52 us; skew 500 ppm
I20260929 15:57:06.945863 22914 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:57:06.946177 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:06.945578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33763 (local address 127.0.0.1:38394)
0929 15:57:06.945902 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:06.946016 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260929 15:57:06.947105 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:06.946748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33763 (local address 127.0.0.1:38410)
0929 15:57:06.946921 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:06.946992 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 15:57:06.947307 22833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33763: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33763: connect: Connection refused (error 111)
Sep 29 15:57:06 dist-test-slave-r0ck krb5kdc[22753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:06.949489 22914 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260929 15:57:06.950963 22914 webserver.cc:460] Webserver started at http://127.0.97.60:41115/ using document root <none> and password file <none>
I20260929 15:57:06.951665 22914 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:06.951751 22914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:06.952103 22914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:06.955035 22914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-2/data/instance:
uuid: "8af6ff3d2afa416d8ec816ff733ff766"
format_stamp: "Formatted at 2026-09-29 15:57:06 on dist-test-slave-r0ck"
server_key: "340138795c80eeaeac23f6c4286e7e94"
server_key_iv: "a9f086a36d9566e750ccb9c7e9bef13e"
server_key_version: "encryptionkey@0"
I20260929 15:57:06.955878 22914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-2/wal/instance:
uuid: "8af6ff3d2afa416d8ec816ff733ff766"
format_stamp: "Formatted at 2026-09-29 15:57:06 on dist-test-slave-r0ck"
server_key: "340138795c80eeaeac23f6c4286e7e94"
server_key_iv: "a9f086a36d9566e750ccb9c7e9bef13e"
server_key_version: "encryptionkey@0"
I20260929 15:57:06.961028 22914 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 15:57:06.964617 22929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:06.966043 22914 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:57:06.966220 22914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-2/wal
uuid: "8af6ff3d2afa416d8ec816ff733ff766"
format_stamp: "Formatted at 2026-09-29 15:57:06 on dist-test-slave-r0ck"
server_key: "340138795c80eeaeac23f6c4286e7e94"
server_key_iv: "a9f086a36d9566e750ccb9c7e9bef13e"
server_key_version: "encryptionkey@0"
I20260929 15:57:06.966418 22914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:06.979949 22914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:06.983944 22914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:06.984254 22914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:07.002780 22914 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:33763
I20260929 15:57:07.002802 22980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:33763 every 8 connection(s)
I20260929 15:57:07.004108 22914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-2/data/info.pb
I20260929 15:57:07.007767 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 22914
I20260929 15:57:07.007980 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697075657932-388-0/minicluster-data/master-2/wal/instance
I20260929 15:57:07.008071 22981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:07.008538 388 external_mini_cluster.cc:1428] Setting key 1e2b125376aac4848609dcee024454be
I20260929 15:57:07.018360 22981 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33163 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:57:07 dist-test-slave-r0ck krb5kdc[22753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 29 15:57:07 dist-test-slave-r0ck krb5kdc[22753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:57:07.034407 22984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:07.010483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38414 (local address 127.0.97.60:33763)
0929 15:57:07.011446 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:07.011458 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:07.011500 (+ 42us) server_negotiation.cc:407] Connection header received
0929 15:57:07.013938 (+ 2438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:07.013962 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:07.014807 (+ 845us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:07.015236 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:07.017786 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.019256 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.021248 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.022427 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.022602 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:07.026978 (+ 4376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:07.026985 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:07.026995 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:07.027029 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:07.030362 (+ 3333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:07.031061 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:07.031067 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:07.031074 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:07.031219 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:07.031735 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:07.031739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:07.031741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:07.032270 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:07.032540 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:07.032969 (+ 429us) server_negotiation.cc:299] Negotiation successful
0929 15:57:07.033319 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":734,"thread_start_us":496,"threads_started":1}
I20260929 15:57:07.036491 22985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:07.021612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33163 (local address 127.0.0.1:43938)
0929 15:57:07.022372 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:07.022408 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:07.022581 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:07.023069 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:07.023079 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:07.023904 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:07.024254 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:07.024269 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.025270 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.025278 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:07.026439 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:07.026451 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.027210 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.027217 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:07.027412 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:07.028138 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:07.028167 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:07.030523 (+ 2356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:07.033900 (+ 3377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:07.033911 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:07.033918 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:07.034357 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:07.034968 (+ 611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:07.034972 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:07.034975 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:07.035132 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:07.035681 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:07.035690 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:07.035857 (+ 167us) client_negotiation.cc:769] Sending connection context
0929 15:57:07.035997 (+ 140us) client_negotiation.cc:240] Negotiation successful
0929 15:57:07.036198 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":419,"threads_started":1}
I20260929 15:57:07.036723 22835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:07.021536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43938 (local address 127.0.97.62:33163)
0929 15:57:07.021727 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:07.021731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:07.022498 (+ 767us) server_negotiation.cc:407] Connection header received
0929 15:57:07.022801 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:07.022808 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:07.022890 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:07.022971 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:07.024412 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.025106 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.026584 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.027014 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.027155 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:07.030720 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:07.030725 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:07.030728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:07.030759 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:07.033768 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:07.034668 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:07.034675 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:07.034681 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:07.034847 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:07.035282 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:07.035285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:07.035287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:07.035543 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:07.035697 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:07.036150 (+ 453us) server_negotiation.cc:299] Negotiation successful
0929 15:57:07.036380 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 15:57:07.038050 22981 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44099 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:57:07 dist-test-slave-r0ck krb5kdc[22753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 29 15:57:07 dist-test-slave-r0ck krb5kdc[22753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:57:07.052453 22985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:07.038786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44099 (local address 127.0.0.1:43176)
0929 15:57:07.038938 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:07.038957 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:07.039046 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:07.039392 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:07.039396 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:07.039726 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:07.039874 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:07.039881 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.040916 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.040921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:07.041929 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:07.041940 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.043273 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.043280 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:07.043500 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:07.044190 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:07.044207 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:07.046324 (+ 2117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:07.050112 (+ 3788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:07.050118 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:07.050122 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:07.050479 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:07.050915 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:07.050919 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:07.050921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:07.051058 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:07.051682 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:07.051685 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:07.051814 (+ 129us) client_negotiation.cc:769] Sending connection context
0929 15:57:07.051929 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 15:57:07.052125 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 15:57:07.052814 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:07.038920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43176 (local address 127.0.97.61:44099)
0929 15:57:07.039089 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:07.039093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:07.039109 (+ 16us) server_negotiation.cc:407] Connection header received
0929 15:57:07.039159 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:07.039162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:07.039217 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:07.039304 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:07.040034 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.040754 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.042225 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.043107 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.043209 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:07.046522 (+ 3313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:07.046528 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:07.046531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:07.046570 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:07.049964 (+ 3394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:07.050618 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:07.050625 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:07.050629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:07.050782 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:07.051180 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:07.051185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:07.051188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:07.051532 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:07.051699 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:07.052067 (+ 368us) server_negotiation.cc:299] Negotiation successful
0929 15:57:07.052277 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 15:57:07.053874 22981 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33763 } has no permanent_uuid. Determining permanent_uuid...
I20260929 15:57:07.055930 22835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:07.037809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43948 (local address 127.0.97.62:33163)
0929 15:57:07.037958 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:07.037964 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:07.038002 (+ 38us) server_negotiation.cc:407] Connection header received
0929 15:57:07.038188 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:07.038193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:07.038263 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:07.038353 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:07.040094 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.040799 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.042940 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.043525 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.043677 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:07.047674 (+ 3997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:07.047680 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:07.047685 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:07.047737 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:07.052217 (+ 4480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:07.053415 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:07.053420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:07.053424 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:07.053582 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:07.053942 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:07.053946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:07.053949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:07.054272 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:07.054557 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:07.055327 (+ 770us) server_negotiation.cc:299] Negotiation successful
0929 15:57:07.055574 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 15:57:07 dist-test-slave-r0ck krb5kdc[22753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 29 15:57:07 dist-test-slave-r0ck krb5kdc[22753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 15:57:07.068819 22985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:07.054606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33763 (local address 127.0.0.1:38418)
0929 15:57:07.054793 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:07.054812 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:07.054903 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:07.055217 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:07.055221 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:07.055677 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:07.055840 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:07.055851 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.057522 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.057528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:07.059115 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:07.059136 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.059868 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.059875 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:07.060107 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:07.060649 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:07.060670 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:07.062980 (+ 2310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:07.066095 (+ 3115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:07.066103 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:07.066110 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:07.066627 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:07.067005 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:07.067010 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:07.067013 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:07.067180 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:07.067806 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:07.067812 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:07.067979 (+ 167us) client_negotiation.cc:769] Sending connection context
0929 15:57:07.068135 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 15:57:07.068534 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 15:57:07.069394 22984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:07.054842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38418 (local address 127.0.97.60:33763)
0929 15:57:07.054990 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:07.054995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:07.055009 (+ 14us) server_negotiation.cc:407] Connection header received
0929 15:57:07.055045 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:07.055050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:07.055107 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:07.055202 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:07.055976 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.057237 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.059295 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.059685 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.059787 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:07.063174 (+ 3387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:07.063180 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:07.063184 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:07.063220 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:07.065964 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:07.066760 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:07.066764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:07.066768 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:07.066887 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:07.067299 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:07.067306 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:07.067312 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:07.067646 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:07.067811 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:07.068428 (+ 617us) server_negotiation.cc:299] Negotiation successful
0929 15:57:07.068840 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 15:57:07.074245 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:07.058019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43180 (local address 127.0.97.61:44099)
0929 15:57:07.058184 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:07.058189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:07.058206 (+ 17us) server_negotiation.cc:407] Connection header received
0929 15:57:07.058358 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:07.058361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:07.058416 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:07.058486 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:07.059908 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.061008 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.062549 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.062972 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.063098 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:07.067162 (+ 4064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:07.067167 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:07.067170 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:07.067198 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:07.071387 (+ 4189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:07.072087 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:07.072094 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:07.072098 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:07.072258 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:07.072792 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:07.072797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:07.072801 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:07.073147 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:07.073430 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:07.073713 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 15:57:07.073936 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 15:57:07.075145 22981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8af6ff3d2afa416d8ec816ff733ff766: Bootstrap starting.
I20260929 15:57:07.078941 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:57:07.079789 22981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8af6ff3d2afa416d8ec816ff733ff766: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:07.080879 22981 log.cc:824] T 00000000000000000000000000000000 P 8af6ff3d2afa416d8ec816ff733ff766: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:07.083657 22981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8af6ff3d2afa416d8ec816ff733ff766: No bootstrap required, opened a new log
I20260929 15:57:07.081589 22808 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "bcef83f3630a40c5a0bdf92c49e56fca"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43948
I20260929 15:57:07.092020 22981 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8af6ff3d2afa416d8ec816ff733ff766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcef83f3630a40c5a0bdf92c49e56fca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33163 } } peers { permanent_uuid: "626d03afb1f348abb00119bb77b7261d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44099 } } peers { permanent_uuid: "8af6ff3d2afa416d8ec816ff733ff766" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33763 } }
I20260929 15:57:07.092422 22981 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8af6ff3d2afa416d8ec816ff733ff766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8af6ff3d2afa416d8ec816ff733ff766, State: Initialized, Role: FOLLOWER
I20260929 15:57:07.093083 22981 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8af6ff3d2afa416d8ec816ff733ff766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcef83f3630a40c5a0bdf92c49e56fca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33163 } } peers { permanent_uuid: "626d03afb1f348abb00119bb77b7261d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44099 } } peers { permanent_uuid: "8af6ff3d2afa416d8ec816ff733ff766" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33763 } }
I20260929 15:57:07.094147 22987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8af6ff3d2afa416d8ec816ff733ff766 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcef83f3630a40c5a0bdf92c49e56fca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33163 } } peers { permanent_uuid: "626d03afb1f348abb00119bb77b7261d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44099 } } peers { permanent_uuid: "8af6ff3d2afa416d8ec816ff733ff766" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33763 } } }
I20260929 15:57:07.094522 22987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8af6ff3d2afa416d8ec816ff733ff766 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:57:07.095130 22981 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8af6ff3d2afa416d8ec816ff733ff766 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 15:57:07.104730 22998 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8af6ff3d2afa416d8ec816ff733ff766: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 15:57:07.104853 22998 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 15:57:07.102368 22914 master_runner.cc:186] Error getting master registration for 127.0.97.62:33163: OK, instance_id { permanent_uuid: "bcef83f3630a40c5a0bdf92c49e56fca" instance_seqno: 1790697425983891 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 15:57:07.110795 22914 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:57:07.124557 22833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33763 } attempt: 4
Sep 29 15:57:07 dist-test-slave-r0ck krb5kdc[22753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 15:57:07.141925 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:07.125391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33763 (local address 127.0.0.1:38424)
0929 15:57:07.125606 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:07.125633 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:07.125766 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:07.126083 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:07.126090 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:07.126598 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:07.126853 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:07.126867 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.128411 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.128417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:07.129926 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:07.129942 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.131011 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.131018 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:07.131206 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:07.131959 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:07.131983 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:07.134874 (+ 2891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:07.139027 (+ 4153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:07.139035 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:07.139041 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:07.139544 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:07.139957 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:07.139962 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:07.139966 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:07.140148 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:07.140842 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:07.140850 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:07.141006 (+ 156us) client_negotiation.cc:769] Sending connection context
0929 15:57:07.141167 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 15:57:07.141474 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 15:57:07.141980 22984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:07.125560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38424 (local address 127.0.97.60:33763)
0929 15:57:07.125760 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:07.125764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:07.125783 (+ 19us) server_negotiation.cc:407] Connection header received
0929 15:57:07.125886 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:07.125890 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:07.125948 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:07.126060 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:07.127000 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.128237 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.130080 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.130846 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.130966 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:07.135089 (+ 4123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:07.135100 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:07.135105 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:07.135150 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:07.138876 (+ 3726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:07.139686 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:07.139694 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:07.139699 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:07.139843 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:07.140276 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:07.140281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:07.140285 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:07.140723 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:07.140878 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:07.141321 (+ 443us) server_negotiation.cc:299] Negotiation successful
0929 15:57:07.141563 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 15:57:07.147992 22833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcef83f3630a40c5a0bdf92c49e56fca: Bootstrap starting.
I20260929 15:57:07.152338 22833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bcef83f3630a40c5a0bdf92c49e56fca: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:07.153744 22833 log.cc:824] T 00000000000000000000000000000000 P bcef83f3630a40c5a0bdf92c49e56fca: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:07.157374 22833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcef83f3630a40c5a0bdf92c49e56fca: No bootstrap required, opened a new log
I20260929 15:57:07.164973 22833 raft_consensus.cc:348] T 00000000000000000000000000000000 P bcef83f3630a40c5a0bdf92c49e56fca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcef83f3630a40c5a0bdf92c49e56fca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33163 } } peers { permanent_uuid: "626d03afb1f348abb00119bb77b7261d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44099 } } peers { permanent_uuid: "8af6ff3d2afa416d8ec816ff733ff766" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33763 } }
I20260929 15:57:07.165417 22833 raft_consensus.cc:729] T 00000000000000000000000000000000 P bcef83f3630a40c5a0bdf92c49e56fca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcef83f3630a40c5a0bdf92c49e56fca, State: Initialized, Role: FOLLOWER
I20260929 15:57:07.165940 22833 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcef83f3630a40c5a0bdf92c49e56fca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcef83f3630a40c5a0bdf92c49e56fca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33163 } } peers { permanent_uuid: "626d03afb1f348abb00119bb77b7261d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44099 } } peers { permanent_uuid: "8af6ff3d2afa416d8ec816ff733ff766" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33763 } }
I20260929 15:57:07.166926 23001 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcef83f3630a40c5a0bdf92c49e56fca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcef83f3630a40c5a0bdf92c49e56fca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33163 } } peers { permanent_uuid: "626d03afb1f348abb00119bb77b7261d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44099 } } peers { permanent_uuid: "8af6ff3d2afa416d8ec816ff733ff766" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33763 } } }
I20260929 15:57:07.167289 23001 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcef83f3630a40c5a0bdf92c49e56fca [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:57:07.168202 22833 sys_catalog.cc:564] T 00000000000000000000000000000000 P bcef83f3630a40c5a0bdf92c49e56fca [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:57:07.168255 22808 raft_consensus.cc:482] T 00000000000000000000000000000000 P bcef83f3630a40c5a0bdf92c49e56fca [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 15:57:07.168458 22808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bcef83f3630a40c5a0bdf92c49e56fca [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:07.171986 22808 raft_consensus.cc:504] T 00000000000000000000000000000000 P bcef83f3630a40c5a0bdf92c49e56fca [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcef83f3630a40c5a0bdf92c49e56fca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33163 } } peers { permanent_uuid: "626d03afb1f348abb00119bb77b7261d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44099 } } peers { permanent_uuid: "8af6ff3d2afa416d8ec816ff733ff766" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33763 } }
I20260929 15:57:07.173651 22808 leader_election.cc:288] T 00000000000000000000000000000000 P bcef83f3630a40c5a0bdf92c49e56fca [CANDIDATE]: Term 1 election: Requested vote from peers 626d03afb1f348abb00119bb77b7261d (127.0.97.61:44099), 8af6ff3d2afa416d8ec816ff733ff766 (127.0.97.60:33763)
I20260929 15:57:07.174584 22956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bcef83f3630a40c5a0bdf92c49e56fca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8af6ff3d2afa416d8ec816ff733ff766"
I20260929 15:57:07.175063 22956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8af6ff3d2afa416d8ec816ff733ff766 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:07.178515 22956 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8af6ff3d2afa416d8ec816ff733ff766 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcef83f3630a40c5a0bdf92c49e56fca in term 1.
I20260929 15:57:07.179144 22784 leader_election.cc:302] T 00000000000000000000000000000000 P bcef83f3630a40c5a0bdf92c49e56fca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8af6ff3d2afa416d8ec816ff733ff766, bcef83f3630a40c5a0bdf92c49e56fca; no voters:
I20260929 15:57:07.179630 23001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bcef83f3630a40c5a0bdf92c49e56fca [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:07.189807 23001 raft_consensus.cc:686] T 00000000000000000000000000000000 P bcef83f3630a40c5a0bdf92c49e56fca [term 1 LEADER]: Becoming Leader. State: Replica: bcef83f3630a40c5a0bdf92c49e56fca, State: Running, Role: LEADER
I20260929 15:57:07.190408 23001 consensus_queue.cc:237] T 00000000000000000000000000000000 P bcef83f3630a40c5a0bdf92c49e56fca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcef83f3630a40c5a0bdf92c49e56fca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33163 } } peers { permanent_uuid: "626d03afb1f348abb00119bb77b7261d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44099 } } peers { permanent_uuid: "8af6ff3d2afa416d8ec816ff733ff766" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33763 } }
I20260929 15:57:07.195052 23016 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcef83f3630a40c5a0bdf92c49e56fca [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcef83f3630a40c5a0bdf92c49e56fca. Latest consensus state: current_term: 1 leader_uuid: "bcef83f3630a40c5a0bdf92c49e56fca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcef83f3630a40c5a0bdf92c49e56fca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33163 } } peers { permanent_uuid: "626d03afb1f348abb00119bb77b7261d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44099 } } peers { permanent_uuid: "8af6ff3d2afa416d8ec816ff733ff766" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33763 } } }
I20260929 15:57:07.195508 23016 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcef83f3630a40c5a0bdf92c49e56fca [sys.catalog]: This master's current role is: LEADER
W20260929 15:57:07.209712 23015 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bcef83f3630a40c5a0bdf92c49e56fca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 15:57:07.209873 23015 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 15:57:07.209985 23018 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:57:07.219393 23018 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:57:07.219740 22907 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33763 } attempt: 5
W20260929 15:57:07.221999 22787 tablet.cc:2369] T 00000000000000000000000000000000 P bcef83f3630a40c5a0bdf92c49e56fca: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 15:57:07 dist-test-slave-r0ck krb5kdc[22753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
W20260929 15:57:07.236598 22935 tablet.cc:2369] T 00000000000000000000000000000000 P 8af6ff3d2afa416d8ec816ff733ff766: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:57:07.246799 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:07.220406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33763 (local address 127.0.0.1:38440)
0929 15:57:07.220858 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:07.220883 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:07.221026 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:07.222169 (+ 1143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:07.222177 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:07.222849 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:07.223133 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:07.223149 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.225026 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.225034 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:07.227010 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:07.227030 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.228523 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.228532 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:07.228761 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:07.232053 (+ 3292us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:07.232079 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:07.235409 (+ 3330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:07.241707 (+ 6298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:07.241734 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:07.241739 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:07.242682 (+ 943us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:07.243384 (+ 702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:07.243411 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:07.243415 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:07.243680 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:07.245043 (+ 1363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:07.245049 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:07.245330 (+ 281us) client_negotiation.cc:769] Sending connection context
0929 15:57:07.245566 (+ 236us) client_negotiation.cc:240] Negotiation successful
0929 15:57:07.246002 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287}
I20260929 15:57:07.202567 22766 master_runner.cc:186] Error getting master registration for 127.0.97.61:44099: OK, instance_id { permanent_uuid: "626d03afb1f348abb00119bb77b7261d" instance_seqno: 1790697426497402 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 15:57:07.248186 22766 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:57:07.248639 22835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:07.211994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43952 (local address 127.0.97.62:33163)
0929 15:57:07.212180 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:07.212186 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:07.226354 (+ 14168us) server_negotiation.cc:407] Connection header received
0929 15:57:07.226746 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:07.226752 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:07.226840 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:07.226914 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:07.230982 (+ 4068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.232108 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.234027 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.234697 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.234808 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:07.240643 (+ 5835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:07.240663 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:07.240668 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:07.240710 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:07.244554 (+ 3844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:07.245438 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:07.245444 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:07.245447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:07.245657 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:07.246161 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:07.246169 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:07.246174 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:07.246593 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:07.247243 (+ 650us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:07.247830 (+ 587us) server_negotiation.cc:299] Negotiation successful
0929 15:57:07.248176 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 15:57:07.252617 22984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:07.221303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38440 (local address 127.0.97.60:33763)
0929 15:57:07.221585 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:07.221594 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:07.221621 (+ 27us) server_negotiation.cc:407] Connection header received
0929 15:57:07.221703 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:07.221711 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:07.221810 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:07.221923 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:07.223348 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.224820 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.227238 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.228192 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.228337 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:07.235594 (+ 7257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:07.235602 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:07.235608 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:07.235647 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:07.241447 (+ 5800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:07.242951 (+ 1504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:07.242957 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:07.242961 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:07.243227 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:07.243907 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:07.243913 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:07.243917 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:07.244706 (+ 789us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:07.247443 (+ 2737us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:07.247970 (+ 527us) server_negotiation.cc:299] Negotiation successful
0929 15:57:07.249031 (+ 1061us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260929 15:57:07.258966 22907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 626d03afb1f348abb00119bb77b7261d: Bootstrap starting.
I20260929 15:57:07.263319 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:07.228942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38442 (local address 127.0.97.60:33763)
0929 15:57:07.237841 (+ 8899us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:07.237848 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:07.238624 (+ 776us) server_negotiation.cc:407] Connection header received
0929 15:57:07.238683 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:07.238689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:07.238773 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:07.238889 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:07.247119 (+ 8230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.248384 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.250375 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.251078 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.251184 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:07.255278 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:07.255288 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:07.255292 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:07.255351 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:07.259359 (+ 4008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:07.260365 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:07.260370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:07.260375 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:07.260554 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:07.260997 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:07.261002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:07.261005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:07.261432 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:07.261674 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:07.262647 (+ 973us) server_negotiation.cc:299] Negotiation successful
0929 15:57:07.262873 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8759,"thread_start_us":9230,"threads_started":1}
I20260929 15:57:07.266541 22956 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8af6ff3d2afa416d8ec816ff733ff766 [term 1 FOLLOWER]: Refusing update from remote peer bcef83f3630a40c5a0bdf92c49e56fca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:57:07.265084 22907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 626d03afb1f348abb00119bb77b7261d: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:07.267666 23001 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bcef83f3630a40c5a0bdf92c49e56fca [LEADER]: Connected to new peer: Peer: permanent_uuid: "8af6ff3d2afa416d8ec816ff733ff766" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:57:07.279928 22907 log.cc:824] T 00000000000000000000000000000000 P 626d03afb1f348abb00119bb77b7261d: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:07.285140 22907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 626d03afb1f348abb00119bb77b7261d: No bootstrap required, opened a new log
I20260929 15:57:07.294970 22987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8af6ff3d2afa416d8ec816ff733ff766 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcef83f3630a40c5a0bdf92c49e56fca. Latest consensus state: current_term: 1 leader_uuid: "bcef83f3630a40c5a0bdf92c49e56fca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcef83f3630a40c5a0bdf92c49e56fca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33163 } } peers { permanent_uuid: "626d03afb1f348abb00119bb77b7261d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44099 } } peers { permanent_uuid: "8af6ff3d2afa416d8ec816ff733ff766" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33763 } } }
I20260929 15:57:07.295411 22987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8af6ff3d2afa416d8ec816ff733ff766 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:57:07.298664 23025 mvcc.cc:205] Tried to move back new op lower bound from 7334696662044569600 to 7334696661777772544. Current Snapshot: MvccSnapshot[applied={T|T < 7334696662044569600}]
I20260929 15:57:07.303803 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:07.211617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43184 (local address 127.0.97.61:44099)
0929 15:57:07.211824 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:07.211834 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:07.230604 (+ 18770us) server_negotiation.cc:407] Connection header received
0929 15:57:07.230749 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:07.230754 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:07.230823 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:07.230914 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:07.274906 (+ 43992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.276458 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.279082 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.279942 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.280098 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:07.294563 (+ 14465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:07.294570 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:07.294575 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:07.294622 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:07.298441 (+ 3819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:07.300530 (+ 2089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:07.300537 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:07.300541 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:07.300706 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:07.301492 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:07.301513 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:07.301516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:07.301925 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:07.302170 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:07.303153 (+ 983us) server_negotiation.cc:299] Negotiation successful
0929 15:57:07.303420 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 15:57:07.323107 22987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8af6ff3d2afa416d8ec816ff733ff766 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bcef83f3630a40c5a0bdf92c49e56fca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcef83f3630a40c5a0bdf92c49e56fca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33163 } } peers { permanent_uuid: "626d03afb1f348abb00119bb77b7261d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44099 } } peers { permanent_uuid: "8af6ff3d2afa416d8ec816ff733ff766" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33763 } } }
I20260929 15:57:07.323359 22987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8af6ff3d2afa416d8ec816ff733ff766 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:57:07.328768 23018 catalog_manager.cc:1347] Generated new cluster ID: 3a29689a8e4e4a81a29c17a893a32875
I20260929 15:57:07.328892 23018 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:57:07.329289 23016 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcef83f3630a40c5a0bdf92c49e56fca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bcef83f3630a40c5a0bdf92c49e56fca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcef83f3630a40c5a0bdf92c49e56fca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33163 } } peers { permanent_uuid: "626d03afb1f348abb00119bb77b7261d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44099 } } peers { permanent_uuid: "8af6ff3d2afa416d8ec816ff733ff766" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33763 } } }
I20260929 15:57:07.329576 23016 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcef83f3630a40c5a0bdf92c49e56fca [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:07.332243 22907 raft_consensus.cc:348] T 00000000000000000000000000000000 P 626d03afb1f348abb00119bb77b7261d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcef83f3630a40c5a0bdf92c49e56fca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33163 } } peers { permanent_uuid: "626d03afb1f348abb00119bb77b7261d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44099 } } peers { permanent_uuid: "8af6ff3d2afa416d8ec816ff733ff766" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33763 } }
I20260929 15:57:07.333681 22907 raft_consensus.cc:729] T 00000000000000000000000000000000 P 626d03afb1f348abb00119bb77b7261d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 626d03afb1f348abb00119bb77b7261d, State: Initialized, Role: FOLLOWER
I20260929 15:57:07.334345 22907 consensus_queue.cc:260] T 00000000000000000000000000000000 P 626d03afb1f348abb00119bb77b7261d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcef83f3630a40c5a0bdf92c49e56fca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33163 } } peers { permanent_uuid: "626d03afb1f348abb00119bb77b7261d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44099 } } peers { permanent_uuid: "8af6ff3d2afa416d8ec816ff733ff766" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33763 } }
I20260929 15:57:07.337658 22907 sys_catalog.cc:564] T 00000000000000000000000000000000 P 626d03afb1f348abb00119bb77b7261d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:57:07.339238 22882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bcef83f3630a40c5a0bdf92c49e56fca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "626d03afb1f348abb00119bb77b7261d"
I20260929 15:57:07.339082 22881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 626d03afb1f348abb00119bb77b7261d [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:07.344436 22881 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 626d03afb1f348abb00119bb77b7261d [term 1 FOLLOWER]: Refusing update from remote peer bcef83f3630a40c5a0bdf92c49e56fca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:57:07.348294 22882 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 626d03afb1f348abb00119bb77b7261d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcef83f3630a40c5a0bdf92c49e56fca in term 1.
I20260929 15:57:07.353850 23031 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bcef83f3630a40c5a0bdf92c49e56fca [LEADER]: Connected to new peer: Peer: permanent_uuid: "626d03afb1f348abb00119bb77b7261d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:57:07.390619 23018 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:57:07.392454 23018 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:57:07.401574 23032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 626d03afb1f348abb00119bb77b7261d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bcef83f3630a40c5a0bdf92c49e56fca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcef83f3630a40c5a0bdf92c49e56fca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33163 } } peers { permanent_uuid: "626d03afb1f348abb00119bb77b7261d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44099 } } peers { permanent_uuid: "8af6ff3d2afa416d8ec816ff733ff766" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33763 } } }
I20260929 15:57:07.401942 23032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 626d03afb1f348abb00119bb77b7261d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:57:07.400965 23037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 626d03afb1f348abb00119bb77b7261d [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcef83f3630a40c5a0bdf92c49e56fca. Latest consensus state: current_term: 1 leader_uuid: "bcef83f3630a40c5a0bdf92c49e56fca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcef83f3630a40c5a0bdf92c49e56fca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33163 } } peers { permanent_uuid: "626d03afb1f348abb00119bb77b7261d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44099 } } peers { permanent_uuid: "8af6ff3d2afa416d8ec816ff733ff766" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33763 } } }
I20260929 15:57:07.402196 23037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 626d03afb1f348abb00119bb77b7261d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:57:07.402551 23044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 626d03afb1f348abb00119bb77b7261d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bcef83f3630a40c5a0bdf92c49e56fca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcef83f3630a40c5a0bdf92c49e56fca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33163 } } peers { permanent_uuid: "626d03afb1f348abb00119bb77b7261d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44099 } } peers { permanent_uuid: "8af6ff3d2afa416d8ec816ff733ff766" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33763 } } }
I20260929 15:57:07.402894 23044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 626d03afb1f348abb00119bb77b7261d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:57:07.404032 23031 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcef83f3630a40c5a0bdf92c49e56fca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bcef83f3630a40c5a0bdf92c49e56fca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcef83f3630a40c5a0bdf92c49e56fca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33163 } } peers { permanent_uuid: "626d03afb1f348abb00119bb77b7261d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44099 } } peers { permanent_uuid: "8af6ff3d2afa416d8ec816ff733ff766" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33763 } } }
I20260929 15:57:07.404405 23031 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcef83f3630a40c5a0bdf92c49e56fca [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:07.420426 23018 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bcef83f3630a40c5a0bdf92c49e56fca: Generated new TSK 0
I20260929 15:57:07.421463 23018 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:57:07.444278 23049 catalog_manager.cc:1259] Loaded cluster ID: 3a29689a8e4e4a81a29c17a893a32875
I20260929 15:57:07.444403 23049 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 626d03afb1f348abb00119bb77b7261d: loading cluster ID for follower catalog manager: success
I20260929 15:57:07.456748 23049 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 626d03afb1f348abb00119bb77b7261d: acquiring CA information for follower catalog manager: success
I20260929 15:57:07.460114 23049 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 626d03afb1f348abb00119bb77b7261d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 15:57:07.569798 22835 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:57:07.551016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37526 (local address 127.0.97.62:33163)
0929 15:57:07.551271 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:07.551277 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:07.562115 (+ 10838us) server_negotiation.cc:407] Connection header received
0929 15:57:07.566216 (+ 4101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:07.566223 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:07.566314 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:07.566393 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:07.569539 (+ 3146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37526: BlockingRecv error: recv got EOF from 127.0.0.1:37526 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260929 15:57:07.571738 22911 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:57:07.552082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60020 (local address 127.0.97.61:44099)
0929 15:57:07.552313 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:07.552320 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:07.568107 (+ 15787us) server_negotiation.cc:407] Connection header received
0929 15:57:07.568765 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:07.568772 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:07.568874 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:07.568966 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:07.571482 (+ 2516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60020: BlockingRecv error: recv got EOF from 127.0.0.1:60020 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260929 15:57:07.574748 23023 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:57:07.562783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50190 (local address 127.0.97.60:33763)
0929 15:57:07.562983 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:07.562989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:07.572598 (+ 9609us) server_negotiation.cc:407] Connection header received
0929 15:57:07.573200 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:07.573206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:07.573293 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:07.573392 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:07.574492 (+ 1100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50190: BlockingRecv error: recv got EOF from 127.0.0.1:50190 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 15:57:07.627799 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:07.611498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60026 (local address 127.0.97.61:44099)
0929 15:57:07.611708 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:07.611715 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:07.619799 (+ 8084us) server_negotiation.cc:407] Connection header received
0929 15:57:07.620371 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:07.620378 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:07.620590 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:07.620732 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:07.621473 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.622634 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.624662 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.625361 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.625460 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:07.626233 (+ 773us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:07.626928 (+ 695us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:07.627045 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:07.627296 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 15:57:07.627398 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260929 15:57:07.682631 23023 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:57:07.671553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50196 (local address 127.0.97.60:33763)
0929 15:57:07.671767 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:07.671774 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:07.680651 (+ 8877us) server_negotiation.cc:407] Connection header received
0929 15:57:07.681122 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:07.681129 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:07.681255 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:07.681327 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:07.682370 (+ 1043us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50196: BlockingRecv error: recv got EOF from 127.0.0.1:50196 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 15:57:07.728193 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:07.719813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60028 (local address 127.0.97.61:44099)
0929 15:57:07.720004 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:07.720010 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:07.721952 (+ 1942us) server_negotiation.cc:407] Connection header received
0929 15:57:07.722177 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:07.722187 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:07.722293 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:07.722399 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:07.722906 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.724125 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.726057 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.726750 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.726890 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:07.727350 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:07.727516 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:07.727610 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:07.727791 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 15:57:07.727884 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 15:57:07.747527 23023 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:57:07.736363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50200 (local address 127.0.97.60:33763)
0929 15:57:07.736570 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:07.736577 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:07.745978 (+ 9401us) server_negotiation.cc:407] Connection header received
0929 15:57:07.746167 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:07.746173 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:07.746271 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:07.746386 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:07.747305 (+ 919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50200: BlockingRecv error: recv got EOF from 127.0.0.1:50200 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 15:57:07.759016 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:07.753074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60042 (local address 127.0.97.61:44099)
0929 15:57:07.753332 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:07.753371 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:07.753920 (+ 549us) server_negotiation.cc:407] Connection header received
0929 15:57:07.754167 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:07.754173 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:07.754252 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:07.754353 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:07.754760 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.755670 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.757162 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.757669 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.757813 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:07.758207 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:07.758308 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:07.758414 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:07.758557 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 15:57:07.758640 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
W20260929 15:57:07.771570 23023 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:57:07.769238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50206 (local address 127.0.97.60:33763)
0929 15:57:07.769493 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:07.769502 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:07.770042 (+ 540us) server_negotiation.cc:407] Connection header received
0929 15:57:07.770298 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:07.770304 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:07.770421 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:07.770517 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:07.771315 (+ 798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50206: BlockingRecv error: recv got EOF from 127.0.0.1:50206 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 15:57:07.784108 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:07.778554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60058 (local address 127.0.97.61:44099)
0929 15:57:07.778845 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:07.778854 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:07.779320 (+ 466us) server_negotiation.cc:407] Connection header received
0929 15:57:07.779525 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:07.779531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:07.779607 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:07.779709 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:07.780112 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.780916 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.782306 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.782772 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.782921 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:07.783311 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:07.783466 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:07.783557 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:07.783713 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 15:57:07.783793 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
W20260929 15:57:07.792176 23023 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:57:07.789955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50208 (local address 127.0.97.60:33763)
0929 15:57:07.790238 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:07.790247 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:07.790750 (+ 503us) server_negotiation.cc:407] Connection header received
0929 15:57:07.790952 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:07.790958 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:07.791060 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:07.791167 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:07.791882 (+ 715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50208: BlockingRecv error: recv got EOF from 127.0.0.1:50208 (error 108)
Metrics: {"server-negotiator.queue_time_us":125}
I20260929 15:57:07.818727 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:07.813242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60070 (local address 127.0.97.61:44099)
0929 15:57:07.813466 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:07.813473 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:07.813811 (+ 338us) server_negotiation.cc:407] Connection header received
0929 15:57:07.814040 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:07.814046 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:07.814116 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:07.814220 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:07.814674 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.815534 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.816828 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.817267 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.817442 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:07.817857 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:07.817984 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:07.818092 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:07.818260 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 15:57:07.818335 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260929 15:57:07.836977 23023 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:57:07.835055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50220 (local address 127.0.97.60:33763)
0929 15:57:07.835235 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:07.835242 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:07.835330 (+ 88us) server_negotiation.cc:407] Connection header received
0929 15:57:07.835638 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:07.835645 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:07.835724 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:07.835834 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:07.836771 (+ 937us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50220: BlockingRecv error: recv got EOF from 127.0.0.1:50220 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 15:57:07.863543 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:07.857386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60078 (local address 127.0.97.61:44099)
0929 15:57:07.857643 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:07.857655 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:07.858239 (+ 584us) server_negotiation.cc:407] Connection header received
0929 15:57:07.858489 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:07.858494 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:07.858566 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:07.858687 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:07.859133 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.860029 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.861526 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.862020 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.862171 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:07.862723 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:07.862830 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:07.862941 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:07.863105 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 15:57:07.863207 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260929 15:57:07.873257 23023 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:57:07.870691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50232 (local address 127.0.97.60:33763)
0929 15:57:07.870978 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:07.870988 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:07.871624 (+ 636us) server_negotiation.cc:407] Connection header received
0929 15:57:07.871855 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:07.871862 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:07.871960 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:07.872093 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:07.873006 (+ 913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50232: BlockingRecv error: recv got EOF from 127.0.0.1:50232 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 15:57:07.914851 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:07.909086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60090 (local address 127.0.97.61:44099)
0929 15:57:07.909296 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:07.909306 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:07.909873 (+ 567us) server_negotiation.cc:407] Connection header received
0929 15:57:07.910101 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:07.910105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:07.910197 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:07.910303 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:07.910727 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.911475 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.912861 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:07.913418 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:07.913561 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:07.913939 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:07.914044 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:07.914160 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:07.914364 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 15:57:07.914447 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 15:57:07.934198 23023 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:57:07.931867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50240 (local address 127.0.97.60:33763)
0929 15:57:07.932129 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:07.932139 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:07.932709 (+ 570us) server_negotiation.cc:407] Connection header received
0929 15:57:07.932938 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:07.932951 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:07.933027 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:07.933135 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:07.933932 (+ 797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50240: BlockingRecv error: recv got EOF from 127.0.0.1:50240 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 15:57:08.011893 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:08.002069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60100 (local address 127.0.97.61:44099)
0929 15:57:08.002253 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:08.002259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:08.006300 (+ 4041us) server_negotiation.cc:407] Connection header received
0929 15:57:08.006495 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:08.006501 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:08.006736 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:08.006861 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:08.007314 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:08.008428 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:08.009821 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:08.010443 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:08.010567 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:08.011066 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:08.011208 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:08.011315 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:08.011509 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 15:57:08.011606 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 15:57:08.022351 23023 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:57:08.018915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50242 (local address 127.0.97.60:33763)
0929 15:57:08.019124 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:08.019131 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:08.020210 (+ 1079us) server_negotiation.cc:407] Connection header received
0929 15:57:08.020512 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:08.020517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:08.020597 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:08.020709 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:08.022128 (+ 1419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50242: BlockingRecv error: recv got EOF from 127.0.0.1:50242 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 15:57:08.107429 22998 catalog_manager.cc:1259] Loaded cluster ID: 3a29689a8e4e4a81a29c17a893a32875
I20260929 15:57:08.107563 22998 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8af6ff3d2afa416d8ec816ff733ff766: loading cluster ID for follower catalog manager: success
I20260929 15:57:08.110476 22998 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8af6ff3d2afa416d8ec816ff733ff766: acquiring CA information for follower catalog manager: success
I20260929 15:57:08.112550 22998 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8af6ff3d2afa416d8ec816ff733ff766: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 15:57:08.160507 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:08.154302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60106 (local address 127.0.97.61:44099)
0929 15:57:08.154555 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:08.154561 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:08.154589 (+ 28us) server_negotiation.cc:407] Connection header received
0929 15:57:08.154872 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:08.154877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:08.154960 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:08.155050 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:08.155969 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:08.156886 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:08.158555 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:08.158982 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:08.159094 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:08.159783 (+ 689us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:08.159912 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:08.159993 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:08.160195 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 15:57:08.160273 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 15:57:08.174117 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:08.167047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50254 (local address 127.0.97.60:33763)
0929 15:57:08.167289 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:08.167294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:08.167565 (+ 271us) server_negotiation.cc:407] Connection header received
0929 15:57:08.167653 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:08.167658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:08.167897 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:08.168015 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:08.168567 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:08.169508 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:08.171043 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:08.171722 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:08.171832 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:08.172545 (+ 713us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:08.173267 (+ 722us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:08.173553 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:08.173697 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 15:57:08.173804 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 15:57:08.189137 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:08.180797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37536 (local address 127.0.97.62:33163)
0929 15:57:08.182009 (+ 1212us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:08.182018 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:08.182269 (+ 251us) server_negotiation.cc:407] Connection header received
0929 15:57:08.182565 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:08.182575 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:08.182752 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:08.182951 (+ 199us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:08.183684 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:08.184415 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:08.186527 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:08.187165 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:08.187302 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:08.187966 (+ 664us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:08.188484 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:08.188589 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:08.188715 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 15:57:08.188800 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":999,"thread_start_us":786,"threads_started":1}
I20260929 15:57:08.189919 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:08.181039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60118 (local address 127.0.97.61:44099)
0929 15:57:08.181311 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:08.181321 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:08.182913 (+ 1592us) server_negotiation.cc:407] Connection header received
0929 15:57:08.183339 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:08.183348 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:08.183461 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:08.183539 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:08.184534 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:08.185645 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:08.187713 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:08.188367 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:08.188493 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:08.188989 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:08.189112 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:08.189207 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:08.189626 (+ 419us) server_negotiation.cc:299] Negotiation successful
0929 15:57:08.189718 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 15:57:08.191787 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:08.183098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50260 (local address 127.0.97.60:33763)
0929 15:57:08.183366 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:08.183374 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:08.185021 (+ 1647us) server_negotiation.cc:407] Connection header received
0929 15:57:08.185215 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:08.185222 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:08.185304 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:08.185431 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:08.186070 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:08.186911 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:08.188640 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:08.189282 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:08.189396 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:08.190912 (+ 1516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:08.191086 (+ 174us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:08.191181 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:08.191354 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 15:57:08.191442 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 15:57:08.194629 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 22766
I20260929 15:57:08.195861 22828 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:08.310827 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 22766
I20260929 15:57:08.333173 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 22840
I20260929 15:57:08.334362 22902 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:08.443950 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 22840
I20260929 15:57:08.464529 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 22914
I20260929 15:57:08.465739 22976 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:08.578397 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 22914
2026-09-29T15:57:08Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4458 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:57:08 dist-test-slave-r0ck krb5kdc[23164](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:57:08 dist-test-slave-r0ck krb5kdc[23164](info): set up 2 sockets
Sep 29 15:57:08 dist-test-slave-r0ck krb5kdc[23164](info): commencing operation
krb5kdc: starting...
W20260929 15:57:09.689991 388 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.006s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/krb5kdc/test-user.keytab.
Sep 29 15:57:09 dist-test-slave-r0ck krb5kdc[23164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:57:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:57:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:57:09.946002 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42789
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42789,127.0.97.61:33081,127.0.97.60:41069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:10.292186 23177 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 15:57:10.292488 23177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:10.292557 23177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:10.292641 23177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:10.300307 23177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:57:10.300406 23177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:10.300660 23177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:10.300791 23177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:57:10.300863 23177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:57:10.312758 23177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42789,127.0.97.61:33081,127.0.97.60:41069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:57:10.314117 23177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:10.315749 23177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:10.329475 23182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:10.329453 23183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:10.329546 23185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:10.330915 23177 server_base.cc:1034] running on GCE node
I20260929 15:57:10.331715 23177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:10.333178 23177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:10.334467 23177 hybrid_clock.cc:648] HybridClock initialized: now 1790697430334407 us; error 53 us; skew 500 ppm
I20260929 15:57:10.334837 23177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:10 dist-test-slave-r0ck krb5kdc[23164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:10.342118 23177 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:57:10.344537 23177 webserver.cc:460] Webserver started at http://127.0.97.62:38231/ using document root <none> and password file <none>
I20260929 15:57:10.345654 23177 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:10.345801 23177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:10.346141 23177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:10.349260 23177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-0/data/instance:
uuid: "a98cb48427e84d8fa212869f63370add"
format_stamp: "Formatted at 2026-09-29 15:57:10 on dist-test-slave-r0ck"
server_key: "3f9678a3b593199ecefe69ba5521d857"
server_key_iv: "f3edb46ce94f386ff47853040a42e397"
server_key_version: "encryptionkey@0"
I20260929 15:57:10.350131 23177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-0/wal/instance:
uuid: "a98cb48427e84d8fa212869f63370add"
format_stamp: "Formatted at 2026-09-29 15:57:10 on dist-test-slave-r0ck"
server_key: "3f9678a3b593199ecefe69ba5521d857"
server_key_iv: "f3edb46ce94f386ff47853040a42e397"
server_key_version: "encryptionkey@0"
I20260929 15:57:10.356197 23177 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 15:57:10.360122 23192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:10.361788 23177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 15:57:10.362000 23177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-0/wal
uuid: "a98cb48427e84d8fa212869f63370add"
format_stamp: "Formatted at 2026-09-29 15:57:10 on dist-test-slave-r0ck"
server_key: "3f9678a3b593199ecefe69ba5521d857"
server_key_iv: "f3edb46ce94f386ff47853040a42e397"
server_key_version: "encryptionkey@0"
I20260929 15:57:10.362227 23177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:10.378297 23177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:10.384575 23177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:10.385030 23177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:10.406666 23177 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42789
I20260929 15:57:10.406634 23243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42789 every 8 connection(s)
I20260929 15:57:10.408475 23177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:57:10.412447 23244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:10.417217 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 23177
I20260929 15:57:10.417476 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-0/wal/instance
I20260929 15:57:10.417912 388 external_mini_cluster.cc:1428] Setting key 15bc52899fb933b4e4d443907f0bf27d
I20260929 15:57:10.419513 23244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42789 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:57:10 dist-test-slave-r0ck krb5kdc[23164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
I20260929 15:57:10.448531 23246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:10.422866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58640 (local address 127.0.97.62:42789)
0929 15:57:10.425921 (+ 3055us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:10.425941 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:10.426554 (+ 613us) server_negotiation.cc:407] Connection header received
0929 15:57:10.428605 (+ 2051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:10.428615 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:10.428887 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:10.429052 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:10.430572 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:10.432532 (+ 1960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:10.434672 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:10.435696 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:10.435921 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:10.439774 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:10.439786 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:10.439807 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:10.439849 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:10.443984 (+ 4135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:10.444762 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:10.444771 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:10.444776 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:10.444963 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:10.445424 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:10.445430 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:10.445433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:10.446169 (+ 736us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:10.446399 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:10.446986 (+ 587us) server_negotiation.cc:299] Negotiation successful
0929 15:57:10.447344 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":2787,"thread_start_us":2543,"threads_started":1}
I20260929 15:57:10.448549 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:10.422866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42789 (local address 127.0.0.1:58640)
0929 15:57:10.426353 (+ 3487us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:10.426401 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:10.428220 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:10.429152 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:10.429161 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:10.430002 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:10.430341 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:10.430357 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:10.432701 (+ 2344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:10.432705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:10.434425 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:10.434453 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:10.436000 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:10.436006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:10.436199 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:10.436966 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:10.436990 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:10.439494 (+ 2504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:10.444166 (+ 4672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:10.444187 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:10.444197 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:10.444585 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:10.445107 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:10.445111 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:10.445113 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:10.445272 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:10.446334 (+ 1062us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:10.446340 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:10.446532 (+ 192us) client_negotiation.cc:769] Sending connection context
0929 15:57:10.446781 (+ 249us) client_negotiation.cc:240] Negotiation successful
0929 15:57:10.447473 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3233,"mutex_wait_us":218,"spinlock_wait_cycles":51968,"thread_start_us":3013,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260929 15:57:10.452138 23244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33081 } has no permanent_uuid. Determining permanent_uuid...
I20260929 15:57:10.453219 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:10.452812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33081 (local address 127.0.0.1:57096)
0929 15:57:10.452994 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:10.453099 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260929 15:57:10.453490 23195 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33081: connect: Connection refused (error 111)
I20260929 15:57:10.455144 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:10.454813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33081 (local address 127.0.0.1:57112)
0929 15:57:10.454957 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:10.455030 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 15:57:10.455338 23244 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33081: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33081: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260929 15:57:10.486725 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:33081
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42789,127.0.97.61:33081,127.0.97.60:41069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:57:10.504655 23244 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33081 } attempt: 1
I20260929 15:57:10.506157 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:10.505488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33081 (local address 127.0.0.1:57124)
0929 15:57:10.505819 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:10.505982 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260929 15:57:10.507426 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:10.507030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33081 (local address 127.0.0.1:57128)
0929 15:57:10.507189 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:10.507271 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 15:57:10.507794 23244 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33081: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33081: connect: Connection refused (error 111)
I20260929 15:57:10.576066 23244 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33081 } attempt: 2
I20260929 15:57:10.577433 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:10.576900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33081 (local address 127.0.0.1:57144)
0929 15:57:10.577165 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:10.577263 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260929 15:57:10.578415 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:10.578062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33081 (local address 127.0.0.1:57158)
0929 15:57:10.578215 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:10.578289 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 15:57:10.578591 23244 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33081: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33081: connect: Connection refused (error 111)
I20260929 15:57:10.669848 23244 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33081 } attempt: 3
I20260929 15:57:10.670948 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:10.670510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33081 (local address 127.0.0.1:57174)
0929 15:57:10.670707 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:10.670814 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 15:57:10.671888 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:10.671544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33081 (local address 127.0.0.1:57188)
0929 15:57:10.671694 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:10.671768 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 15:57:10.672070 23244 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33081: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33081: connect: Connection refused (error 111)
I20260929 15:57:10.815335 23244 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33081 } attempt: 4
I20260929 15:57:10.816617 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:10.816121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33081 (local address 127.0.0.1:57200)
0929 15:57:10.816413 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:10.816494 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260929 15:57:10.817565 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:10.817190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33081 (local address 127.0.0.1:57216)
0929 15:57:10.817372 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:10.817456 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 15:57:10.817724 23244 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33081: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33081: connect: Connection refused (error 111)
W20260929 15:57:10.849241 23251 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 15:57:10.850076 23251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:10.850217 23251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:10.850374 23251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:10.859191 23251 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:57:10.859335 23251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:10.859467 23251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:10.859560 23251 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:57:10.859671 23251 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:57:10.871716 23251 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42789,127.0.97.61:33081,127.0.97.60:41069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:33081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:57:10.873111 23251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:10.874481 23251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:10.887874 23259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:10.887965 23257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:10.888245 23256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:10.889016 23251 server_base.cc:1034] running on GCE node
I20260929 15:57:10.890023 23251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:10.891558 23251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:10.892803 23251 hybrid_clock.cc:648] HybridClock initialized: now 1790697430892742 us; error 61 us; skew 500 ppm
I20260929 15:57:10.893146 23251 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:10 dist-test-slave-r0ck krb5kdc[23164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:10.896935 23251 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260929 15:57:10.899119 23251 webserver.cc:460] Webserver started at http://127.0.97.61:39219/ using document root <none> and password file <none>
I20260929 15:57:10.900125 23251 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:10.900316 23251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:10.900661 23251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:10.903796 23251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-1/data/instance:
uuid: "9e39f91d0da842769c276bd202d468c4"
format_stamp: "Formatted at 2026-09-29 15:57:10 on dist-test-slave-r0ck"
server_key: "2c464da10f4bd0a660588c1c0fcdcf59"
server_key_iv: "96efd44f0fc55ccf78e5e0717d8c8b3c"
server_key_version: "encryptionkey@0"
I20260929 15:57:10.904608 23251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-1/wal/instance:
uuid: "9e39f91d0da842769c276bd202d468c4"
format_stamp: "Formatted at 2026-09-29 15:57:10 on dist-test-slave-r0ck"
server_key: "2c464da10f4bd0a660588c1c0fcdcf59"
server_key_iv: "96efd44f0fc55ccf78e5e0717d8c8b3c"
server_key_version: "encryptionkey@0"
I20260929 15:57:10.909988 23251 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 15:57:10.913831 23266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:10.915447 23251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 15:57:10.915657 23251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-1/wal
uuid: "9e39f91d0da842769c276bd202d468c4"
format_stamp: "Formatted at 2026-09-29 15:57:10 on dist-test-slave-r0ck"
server_key: "2c464da10f4bd0a660588c1c0fcdcf59"
server_key_iv: "96efd44f0fc55ccf78e5e0717d8c8b3c"
server_key_version: "encryptionkey@0"
I20260929 15:57:10.915894 23251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:10.931663 23251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:10.937515 23251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:10.937810 23251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:10.958441 23251 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:33081
I20260929 15:57:10.958485 23317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:33081 every 8 connection(s)
I20260929 15:57:10.959789 23251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-1/data/info.pb
I20260929 15:57:10.963554 23318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:10.963601 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 23251
I20260929 15:57:10.963778 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-1/wal/instance
I20260929 15:57:10.964346 388 external_mini_cluster.cc:1428] Setting key 066c678b2561fa8c4a72a63625e7e573
I20260929 15:57:10.973527 23318 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42789 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:57:10 dist-test-slave-r0ck krb5kdc[23164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260929 15:57:10.994343 23321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:10.976716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58650 (local address 127.0.97.62:42789)
0929 15:57:10.977210 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:10.977215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:10.978229 (+ 1014us) server_negotiation.cc:407] Connection header received
0929 15:57:10.980095 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:10.980101 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:10.980197 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:10.980328 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:10.981881 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:10.982594 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:10.984083 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:10.984500 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:10.984635 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:10.988292 (+ 3657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:10.988304 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:10.988308 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:10.988350 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:10.991124 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:10.991838 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:10.991847 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:10.991849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:10.992005 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:10.992428 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:10.992433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:10.992436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:10.992709 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:10.992868 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:10.993743 (+ 875us) server_negotiation.cc:299] Negotiation successful
0929 15:57:10.993955 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":274,"threads_started":1}
I20260929 15:57:10.995234 23322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:10.977287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42789 (local address 127.0.0.1:58650)
0929 15:57:10.978047 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:10.978104 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:10.979829 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:10.980557 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:10.980566 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:10.981074 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:10.981675 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:10.981692 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:10.982769 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:10.982773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:10.983927 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:10.983939 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:10.984734 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:10.984741 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:10.984967 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:10.985730 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:10.985757 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:10.988059 (+ 2302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:10.991260 (+ 3201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:10.991270 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:10.991286 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:10.991696 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:10.992116 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:10.992121 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:10.992125 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:10.992307 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:10.992842 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:10.992849 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:10.993268 (+ 419us) client_negotiation.cc:769] Sending connection context
0929 15:57:10.993640 (+ 372us) client_negotiation.cc:240] Negotiation successful
0929 15:57:10.994147 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":326,"threads_started":1}
I20260929 15:57:10.997658 23318 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33081 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:57:11 dist-test-slave-r0ck krb5kdc[23164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 15:57:11.014964 23322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:10.998631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33081 (local address 127.0.0.1:57228)
0929 15:57:10.998877 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:10.998892 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:10.998974 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:11.000273 (+ 1299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:11.000277 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:11.000707 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:11.000825 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:11.000833 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:11.002288 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:11.002294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:11.003226 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:11.003239 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:11.004064 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:11.004070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:11.004369 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:11.005714 (+ 1345us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:11.005733 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:11.007780 (+ 2047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:11.011071 (+ 3291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:11.011078 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:11.011084 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:11.011444 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:11.012133 (+ 689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:11.012140 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:11.012144 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:11.012289 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:11.012897 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:11.012909 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:11.013628 (+ 719us) client_negotiation.cc:769] Sending connection context
0929 15:57:11.013925 (+ 297us) client_negotiation.cc:240] Negotiation successful
0929 15:57:11.014370 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":54}
I20260929 15:57:11.015129 23323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:10.999026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57228 (local address 127.0.97.61:33081)
0929 15:57:10.999871 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:10.999878 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:10.999902 (+ 24us) server_negotiation.cc:407] Connection header received
0929 15:57:10.999941 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:10.999947 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:11.000103 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:11.000279 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:11.001006 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:11.002132 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:11.003378 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:11.003896 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:11.004045 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:11.007996 (+ 3951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:11.008008 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:11.008022 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:11.008060 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:11.010917 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:11.011855 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:11.011875 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:11.011877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:11.012006 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:11.012405 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:11.012410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:11.012420 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:11.012769 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:11.013150 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:11.014143 (+ 993us) server_negotiation.cc:299] Negotiation successful
0929 15:57:11.014571 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":202,"server-negotiator.queue_time_us":681,"thread_start_us":545,"threads_started":1}
I20260929 15:57:11.018399 23318 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41069 } has no permanent_uuid. Determining permanent_uuid...
I20260929 15:57:11.019794 23322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:11.019184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41069 (local address 127.0.0.1:41410)
0929 15:57:11.019513 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:11.019631 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260929 15:57:11.020078 23269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41069: connect: Connection refused (error 111)
I20260929 15:57:11.021790 23322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:11.021456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41069 (local address 127.0.0.1:41414)
0929 15:57:11.021597 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:11.021670 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 15:57:11.022025 23318 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41069: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41069: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260929 15:57:11.034806 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskorfijW/build/asan/bin/kudu
/tmp/dist-test-taskorfijW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:41069
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42789,127.0.97.61:33081,127.0.97.60:41069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:57:11.071408 23318 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41069 } attempt: 1
I20260929 15:57:11.072988 23322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:11.072314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41069 (local address 127.0.0.1:41420)
0929 15:57:11.072627 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:11.072778 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260929 15:57:11.074239 23322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:11.073829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41069 (local address 127.0.0.1:41422)
0929 15:57:11.074014 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:11.074102 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260929 15:57:11.074512 23318 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41069: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41069: connect: Connection refused (error 111)
I20260929 15:57:11.116995 23244 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33081 } attempt: 5
Sep 29 15:57:11 dist-test-slave-r0ck krb5kdc[23164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 15:57:11.132488 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:11.117845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33081 (local address 127.0.0.1:57234)
0929 15:57:11.118062 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:11.118083 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:11.118228 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:11.118602 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:11.118606 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:11.118977 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:11.119183 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:11.119197 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:11.120822 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:11.120827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:11.122061 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:11.122078 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:11.123122 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:11.123126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:11.123246 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:11.124035 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:11.124053 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:11.126232 (+ 2179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:11.130296 (+ 4064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:11.130300 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:11.130303 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:11.130667 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:11.131092 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:11.131095 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:11.131097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:11.131219 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:11.131791 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:11.131794 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:11.131906 (+ 112us) client_negotiation.cc:769] Sending connection context
0929 15:57:11.132031 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 15:57:11.132247 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 15:57:11.132673 23323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:11.117995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57234 (local address 127.0.97.61:33081)
0929 15:57:11.118232 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:11.118238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:11.118261 (+ 23us) server_negotiation.cc:407] Connection header received
0929 15:57:11.118346 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:11.118351 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:11.118431 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:11.118546 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:11.119403 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:11.120598 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:11.122241 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:11.122942 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:11.123076 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:11.126426 (+ 3350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:11.126433 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:11.126437 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:11.126474 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:11.130135 (+ 3661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:11.130799 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:11.130805 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:11.130811 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:11.130970 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:11.131322 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:11.131327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:11.131330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:11.131666 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:11.131810 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:11.132163 (+ 353us) server_negotiation.cc:299] Negotiation successful
0929 15:57:11.132385 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 15:57:11.133942 23244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41069 } has no permanent_uuid. Determining permanent_uuid...
I20260929 15:57:11.135082 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:11.134707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41069 (local address 127.0.0.1:41436)
0929 15:57:11.134885 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:11.134956 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260929 15:57:11.136265 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:11.135888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41069 (local address 127.0.0.1:41446)
0929 15:57:11.136057 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:11.136165 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260929 15:57:11.136591 23244 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41069: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41069: connect: Connection refused (error 111)
I20260929 15:57:11.142774 23318 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41069 } attempt: 2
I20260929 15:57:11.143709 23322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:11.143285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41069 (local address 127.0.0.1:41458)
0929 15:57:11.143459 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:11.143556 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 15:57:11.144831 23322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:11.144478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41069 (local address 127.0.0.1:41468)
0929 15:57:11.144624 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:11.144702 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 15:57:11.145052 23318 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41069: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41069: connect: Connection refused (error 111)
I20260929 15:57:11.187918 23244 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41069 } attempt: 1
I20260929 15:57:11.189807 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:11.189020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41069 (local address 127.0.0.1:41470)
0929 15:57:11.189329 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:11.189547 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260929 15:57:11.190887 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:11.190565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41069 (local address 127.0.0.1:41482)
0929 15:57:11.190706 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:11.190788 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 15:57:11.191089 23244 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41069: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41069: connect: Connection refused (error 111)
I20260929 15:57:11.236326 23318 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41069 } attempt: 3
I20260929 15:57:11.237709 23322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:11.237117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41069 (local address 127.0.0.1:41498)
0929 15:57:11.237411 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:11.237523 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260929 15:57:11.238902 23322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:11.238372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41069 (local address 127.0.0.1:41510)
0929 15:57:11.238675 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:11.238765 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205}
W20260929 15:57:11.239262 23318 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41069: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41069: connect: Connection refused (error 111)
I20260929 15:57:11.259351 23244 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41069 } attempt: 2
I20260929 15:57:11.260502 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:11.260023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41069 (local address 127.0.0.1:41516)
0929 15:57:11.260260 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:11.260339 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 15:57:11.261636 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:11.261248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41069 (local address 127.0.0.1:41528)
0929 15:57:11.261453 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:11.261530 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 15:57:11.261945 23244 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41069: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41069: connect: Connection refused (error 111)
I20260929 15:57:11.368197 23244 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41069 } attempt: 3
I20260929 15:57:11.369393 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:11.368926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41069 (local address 127.0.0.1:41534)
0929 15:57:11.369118 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:11.369203 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 15:57:11.370347 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:11.370019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41069 (local address 127.0.0.1:41546)
0929 15:57:11.370174 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:11.370258 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260929 15:57:11.370612 23244 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41069: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41069: connect: Connection refused (error 111)
W20260929 15:57:11.373085 23326 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 15:57:11.373499 23326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:11.373636 23326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:11.373767 23326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:11.381163 23326 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:57:11.381247 23326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:11.381305 23326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:11.381412 23326 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:57:11.381494 23326 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:57:11.382620 23318 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41069 } attempt: 4
I20260929 15:57:11.383713 23322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:11.383218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41069 (local address 127.0.0.1:41558)
0929 15:57:11.383440 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:11.383532 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 15:57:11.384895 23322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:11.384424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41069 (local address 127.0.0.1:41568)
0929 15:57:11.384668 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:11.384768 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
W20260929 15:57:11.385231 23318 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41069: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41069: connect: Connection refused (error 111)
I20260929 15:57:11.392462 23326 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42789,127.0.97.61:33081,127.0.97.60:41069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:41069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:57:11.393838 23326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:11.395099 23326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:11.407739 23332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:11.407756 23334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:11.408601 23326 server_base.cc:1034] running on GCE node
W20260929 15:57:11.409018 23331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:11.410394 23326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:11.412263 23326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:11.413607 23326 hybrid_clock.cc:648] HybridClock initialized: now 1790697431413529 us; error 59 us; skew 500 ppm
I20260929 15:57:11.414008 23326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:11 dist-test-slave-r0ck krb5kdc[23164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:11.417838 23326 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260929 15:57:11.419345 23326 webserver.cc:460] Webserver started at http://127.0.97.60:37759/ using document root <none> and password file <none>
I20260929 15:57:11.420326 23326 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:11.420440 23326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:11.420804 23326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:11.423952 23326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-2/data/instance:
uuid: "f484056955bb4eea8ee153f1aae070fc"
format_stamp: "Formatted at 2026-09-29 15:57:11 on dist-test-slave-r0ck"
server_key: "35ccc2d31c10baaa00cc86505de9bedf"
server_key_iv: "22f283864d8cdc057acd40812f43dbf6"
server_key_version: "encryptionkey@0"
I20260929 15:57:11.424969 23326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-2/wal/instance:
uuid: "f484056955bb4eea8ee153f1aae070fc"
format_stamp: "Formatted at 2026-09-29 15:57:11 on dist-test-slave-r0ck"
server_key: "35ccc2d31c10baaa00cc86505de9bedf"
server_key_iv: "22f283864d8cdc057acd40812f43dbf6"
server_key_version: "encryptionkey@0"
I20260929 15:57:11.431910 23326 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 15:57:11.436177 23341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:11.437950 23326 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 15:57:11.438155 23326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-2/wal
uuid: "f484056955bb4eea8ee153f1aae070fc"
format_stamp: "Formatted at 2026-09-29 15:57:11 on dist-test-slave-r0ck"
server_key: "35ccc2d31c10baaa00cc86505de9bedf"
server_key_iv: "22f283864d8cdc057acd40812f43dbf6"
server_key_version: "encryptionkey@0"
I20260929 15:57:11.438385 23326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:11.468055 23326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:11.473246 23326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:11.473603 23326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:11.493302 23326 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:41069
I20260929 15:57:11.493314 23392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:41069 every 8 connection(s)
I20260929 15:57:11.495062 23326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-2/data/info.pb
I20260929 15:57:11.498632 23393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:11.501160 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskorfijW/build/asan/bin/kudu as pid 23326
I20260929 15:57:11.501459 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskorfijW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697075657932-388-0/minicluster-data/master-2/wal/instance
I20260929 15:57:11.502007 388 external_mini_cluster.cc:1428] Setting key 1fe6e8f9363a90802ae6ac7a77c394f5
I20260929 15:57:11.506491 23393 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42789 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:57:11 dist-test-slave-r0ck krb5kdc[23164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 29 15:57:11 dist-test-slave-r0ck krb5kdc[23164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:57:11.525799 23397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:11.509307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58652 (local address 127.0.97.62:42789)
0929 15:57:11.509879 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:11.509885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:11.510366 (+ 481us) server_negotiation.cc:407] Connection header received
0929 15:57:11.510677 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:11.510683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:11.510805 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:11.511021 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:11.512559 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:11.513467 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:11.514884 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:11.515306 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:11.515653 (+ 347us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:11.519374 (+ 3721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:11.519379 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:11.519382 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:11.519414 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:11.523071 (+ 3657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:11.523832 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:11.523840 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:11.523844 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:11.523971 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:11.524336 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:11.524343 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:11.524346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:11.524667 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:11.524820 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:11.525304 (+ 484us) server_negotiation.cc:299] Negotiation successful
0929 15:57:11.525522 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":242,"threads_started":1}
I20260929 15:57:11.526307 23398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:11.509305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42789 (local address 127.0.0.1:58652)
0929 15:57:11.510214 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:11.510258 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:11.510416 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:11.511148 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:11.511162 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:11.512006 (+ 844us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:11.512374 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:11.512386 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:11.513622 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:11.513634 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:11.514712 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:11.514724 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:11.515736 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:11.515745 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:11.515947 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:11.516611 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:11.516654 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:11.519153 (+ 2499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:11.523200 (+ 4047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:11.523212 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:11.523219 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:11.523703 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:11.524090 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:11.524096 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:11.524105 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:11.524221 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:11.524811 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:11.524819 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:11.525017 (+ 198us) client_negotiation.cc:769] Sending connection context
0929 15:57:11.525201 (+ 184us) client_negotiation.cc:240] Negotiation successful
0929 15:57:11.525522 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":501,"threads_started":1}
I20260929 15:57:11.526328 23396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:11.503850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41584 (local address 127.0.97.60:41069)
0929 15:57:11.504631 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:11.504642 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:11.504686 (+ 44us) server_negotiation.cc:407] Connection header received
0929 15:57:11.506420 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:11.506441 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:11.506850 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:11.507163 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:11.508859 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:11.510822 (+ 1963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:11.512064 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:11.512748 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:11.512935 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:11.517839 (+ 4904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:11.517850 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:11.517865 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:11.517907 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:11.521276 (+ 3369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:11.522257 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:11.522263 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:11.522269 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:11.522419 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:11.522954 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:11.522958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:11.522960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:11.523480 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:11.523909 (+ 429us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:11.524371 (+ 462us) server_negotiation.cc:299] Negotiation successful
0929 15:57:11.524968 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":373,"threads_started":1}
I20260929 15:57:11.528394 23393 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33081 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:57:11 dist-test-slave-r0ck krb5kdc[23164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 29 15:57:11 dist-test-slave-r0ck krb5kdc[23164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:57:11.543869 23398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:11.529166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33081 (local address 127.0.0.1:57240)
0929 15:57:11.529544 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:11.529563 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:11.529670 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:11.530066 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:11.530071 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:11.530551 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:11.530746 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:11.530758 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:11.531826 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:11.531833 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:11.533073 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:11.533084 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:11.534300 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:11.534306 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:11.534489 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:11.535141 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:11.535163 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:11.537500 (+ 2337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:11.541230 (+ 3730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:11.541236 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:11.541240 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:11.541749 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:11.542127 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:11.542133 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:11.542136 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:11.542285 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:11.542917 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:11.542923 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:11.543079 (+ 156us) client_negotiation.cc:769] Sending connection context
0929 15:57:11.543214 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 15:57:11.543493 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262}
I20260929 15:57:11.543890 23323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:11.529463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57240 (local address 127.0.97.61:33081)
0929 15:57:11.529628 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:11.529632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:11.529649 (+ 17us) server_negotiation.cc:407] Connection header received
0929 15:57:11.529805 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:11.529811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:11.529903 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:11.530004 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:11.530901 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:11.531651 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:11.533279 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:11.534111 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:11.534245 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:11.537700 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:11.537706 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:11.537709 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:11.537741 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:11.541079 (+ 3338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:11.541873 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:11.541885 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:11.541888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:11.542013 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:11.542454 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:11.542466 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:11.542469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:11.542797 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:11.542954 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:11.543316 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 15:57:11.543522 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 15:57:11.544883 23393 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41069 } has no permanent_uuid. Determining permanent_uuid...
I20260929 15:57:11.546984 23397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:11.530269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58660 (local address 127.0.97.62:42789)
0929 15:57:11.530565 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:11.530570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:11.530594 (+ 24us) server_negotiation.cc:407] Connection header received
0929 15:57:11.530759 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:11.530763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:11.530881 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:11.530994 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:11.532547 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:11.533636 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:11.535341 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:11.535837 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:11.535939 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:11.540665 (+ 4726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:11.540670 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:11.540673 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:11.540709 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:11.543546 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:11.544438 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:11.544442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:11.544445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:11.544591 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:11.544986 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:11.544990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:11.544993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:11.545335 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:11.545645 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:11.546424 (+ 779us) server_negotiation.cc:299] Negotiation successful
0929 15:57:11.546641 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260929 15:57:11.547834 23244 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41069 } attempt: 4
Sep 29 15:57:11 dist-test-slave-r0ck krb5kdc[23164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 29 15:57:11 dist-test-slave-r0ck krb5kdc[23164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 29 15:57:11 dist-test-slave-r0ck krb5kdc[23164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 15:57:11.562378 23396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:11.545846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41596 (local address 127.0.97.60:41069)
0929 15:57:11.546207 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:11.546213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:11.546261 (+ 48us) server_negotiation.cc:407] Connection header received
0929 15:57:11.546339 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:11.546363 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:11.546472 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:11.546598 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:11.547404 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:11.548265 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:11.549926 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:11.550652 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:11.550817 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:11.554552 (+ 3735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:11.554557 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:11.554561 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:11.554589 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:11.558698 (+ 4109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:11.559746 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:11.559754 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:11.559761 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:11.559997 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:11.560638 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:11.560645 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:11.560648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:11.561103 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:11.561251 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:11.561752 (+ 501us) server_negotiation.cc:299] Negotiation successful
0929 15:57:11.561979 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":198}
I20260929 15:57:11.562572 23398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:11.545654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41069 (local address 127.0.0.1:41596)
0929 15:57:11.545885 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:11.545903 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:11.545985 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:11.546648 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:11.546652 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:11.547085 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:11.547260 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:11.547269 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:11.548384 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:11.548392 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:11.549776 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:11.549791 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:11.551055 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:11.551062 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:11.551322 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:11.552041 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:11.552062 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:11.554355 (+ 2293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:11.558870 (+ 4515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:11.558874 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:11.558879 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:11.559589 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:11.560198 (+ 609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:11.560204 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:11.560210 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:11.560495 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:11.561303 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:11.561307 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:11.561505 (+ 198us) client_negotiation.cc:769] Sending connection context
0929 15:57:11.561674 (+ 169us) client_negotiation.cc:240] Negotiation successful
0929 15:57:11.562156 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":148,"spinlock_wait_cycles":33152}
I20260929 15:57:11.567943 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:11.548493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41069 (local address 127.0.0.1:41604)
0929 15:57:11.548679 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:11.548699 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:11.548796 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:11.550547 (+ 1751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:11.550555 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:11.551074 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:11.551246 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:11.551255 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:11.553060 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:11.553069 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:11.554618 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:11.554634 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:11.555648 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:11.555654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:11.555880 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:11.556538 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:11.556570 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:11.559948 (+ 3378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:11.565178 (+ 5230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:11.565185 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:11.565189 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:11.565777 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:11.566191 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:11.566199 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:11.566203 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:11.566356 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:11.566984 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:11.566989 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:11.567142 (+ 153us) client_negotiation.cc:769] Sending connection context
0929 15:57:11.567285 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 15:57:11.567580 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 15:57:11.568174 23323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:11.549754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57250 (local address 127.0.97.61:33081)
0929 15:57:11.549982 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:11.549989 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:11.550014 (+ 25us) server_negotiation.cc:407] Connection header received
0929 15:57:11.550149 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:11.550154 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:11.550243 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:11.550309 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:11.552085 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:11.553200 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:11.555085 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:11.555859 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:11.556048 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:11.561515 (+ 5467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:11.561525 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:11.561530 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:11.561574 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:11.564779 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:11.565582 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:11.565587 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:11.565591 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:11.565806 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:11.566307 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:11.566313 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:11.566316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:11.566703 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:11.567057 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:11.567442 (+ 385us) server_negotiation.cc:299] Negotiation successful
0929 15:57:11.567662 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 15:57:11.568895 23399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:11.548818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41604 (local address 127.0.97.60:41069)
0929 15:57:11.550237 (+ 1419us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:11.550241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:11.550258 (+ 17us) server_negotiation.cc:407] Connection header received
0929 15:57:11.550297 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:11.550300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:11.550372 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:11.550513 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:11.551435 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:11.552890 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:11.554774 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:11.555486 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:11.555631 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:11.560256 (+ 4625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:11.560263 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:11.560270 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:11.560380 (+ 110us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:11.565033 (+ 4653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:11.565897 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:11.565902 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:11.565906 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:11.566062 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:11.566476 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:11.566483 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:11.566486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:11.566825 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:11.568204 (+ 1379us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:11.568345 (+ 141us) server_negotiation.cc:299] Negotiation successful
0929 15:57:11.568547 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":1305,"thread_start_us":1173,"threads_started":1}
I20260929 15:57:11.569923 23393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f484056955bb4eea8ee153f1aae070fc: Bootstrap starting.
I20260929 15:57:11.573148 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:57:11.573802 23393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f484056955bb4eea8ee153f1aae070fc: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:11.573864 23244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a98cb48427e84d8fa212869f63370add: Bootstrap starting.
I20260929 15:57:11.574815 23393 log.cc:824] T 00000000000000000000000000000000 P f484056955bb4eea8ee153f1aae070fc: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:11.577379 23244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a98cb48427e84d8fa212869f63370add: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:11.578296 23393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f484056955bb4eea8ee153f1aae070fc: No bootstrap required, opened a new log
I20260929 15:57:11.578838 23244 log.cc:824] T 00000000000000000000000000000000 P a98cb48427e84d8fa212869f63370add: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:11.576192 23219 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a98cb48427e84d8fa212869f63370add"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58660
I20260929 15:57:11.582629 23244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a98cb48427e84d8fa212869f63370add: No bootstrap required, opened a new log
I20260929 15:57:11.590867 23244 raft_consensus.cc:348] T 00000000000000000000000000000000 P a98cb48427e84d8fa212869f63370add [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98cb48427e84d8fa212869f63370add" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42789 } } peers { permanent_uuid: "9e39f91d0da842769c276bd202d468c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33081 } } peers { permanent_uuid: "f484056955bb4eea8ee153f1aae070fc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41069 } }
I20260929 15:57:11.590732 23393 raft_consensus.cc:348] T 00000000000000000000000000000000 P f484056955bb4eea8ee153f1aae070fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98cb48427e84d8fa212869f63370add" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42789 } } peers { permanent_uuid: "9e39f91d0da842769c276bd202d468c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33081 } } peers { permanent_uuid: "f484056955bb4eea8ee153f1aae070fc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41069 } }
I20260929 15:57:11.591282 23393 raft_consensus.cc:729] T 00000000000000000000000000000000 P f484056955bb4eea8ee153f1aae070fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f484056955bb4eea8ee153f1aae070fc, State: Initialized, Role: FOLLOWER
I20260929 15:57:11.591276 23244 raft_consensus.cc:729] T 00000000000000000000000000000000 P a98cb48427e84d8fa212869f63370add [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a98cb48427e84d8fa212869f63370add, State: Initialized, Role: FOLLOWER
I20260929 15:57:11.591878 23244 consensus_queue.cc:260] T 00000000000000000000000000000000 P a98cb48427e84d8fa212869f63370add [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98cb48427e84d8fa212869f63370add" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42789 } } peers { permanent_uuid: "9e39f91d0da842769c276bd202d468c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33081 } } peers { permanent_uuid: "f484056955bb4eea8ee153f1aae070fc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41069 } }
I20260929 15:57:11.591889 23393 consensus_queue.cc:260] T 00000000000000000000000000000000 P f484056955bb4eea8ee153f1aae070fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98cb48427e84d8fa212869f63370add" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42789 } } peers { permanent_uuid: "9e39f91d0da842769c276bd202d468c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33081 } } peers { permanent_uuid: "f484056955bb4eea8ee153f1aae070fc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41069 } }
I20260929 15:57:11.592995 23402 sys_catalog.cc:455] T 00000000000000000000000000000000 P f484056955bb4eea8ee153f1aae070fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98cb48427e84d8fa212869f63370add" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42789 } } peers { permanent_uuid: "9e39f91d0da842769c276bd202d468c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33081 } } peers { permanent_uuid: "f484056955bb4eea8ee153f1aae070fc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41069 } } }
I20260929 15:57:11.593483 23402 sys_catalog.cc:458] T 00000000000000000000000000000000 P f484056955bb4eea8ee153f1aae070fc [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:57:11.594149 23403 sys_catalog.cc:455] T 00000000000000000000000000000000 P a98cb48427e84d8fa212869f63370add [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98cb48427e84d8fa212869f63370add" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42789 } } peers { permanent_uuid: "9e39f91d0da842769c276bd202d468c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33081 } } peers { permanent_uuid: "f484056955bb4eea8ee153f1aae070fc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41069 } } }
I20260929 15:57:11.594587 23403 sys_catalog.cc:458] T 00000000000000000000000000000000 P a98cb48427e84d8fa212869f63370add [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:57:11.594828 23393 sys_catalog.cc:564] T 00000000000000000000000000000000 P f484056955bb4eea8ee153f1aae070fc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:57:11.595300 23244 sys_catalog.cc:564] T 00000000000000000000000000000000 P a98cb48427e84d8fa212869f63370add [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:57:11.595450 23219 raft_consensus.cc:482] T 00000000000000000000000000000000 P a98cb48427e84d8fa212869f63370add [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 15:57:11.595669 23219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a98cb48427e84d8fa212869f63370add [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:11.599309 23219 raft_consensus.cc:504] T 00000000000000000000000000000000 P a98cb48427e84d8fa212869f63370add [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98cb48427e84d8fa212869f63370add" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42789 } } peers { permanent_uuid: "9e39f91d0da842769c276bd202d468c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33081 } } peers { permanent_uuid: "f484056955bb4eea8ee153f1aae070fc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41069 } }
I20260929 15:57:11.602173 23219 leader_election.cc:288] T 00000000000000000000000000000000 P a98cb48427e84d8fa212869f63370add [CANDIDATE]: Term 1 election: Requested vote from peers 9e39f91d0da842769c276bd202d468c4 (127.0.97.61:33081), f484056955bb4eea8ee153f1aae070fc (127.0.97.60:41069)
I20260929 15:57:11.606364 23368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a98cb48427e84d8fa212869f63370add" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f484056955bb4eea8ee153f1aae070fc"
I20260929 15:57:11.606845 23368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f484056955bb4eea8ee153f1aae070fc [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:11.610288 23368 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f484056955bb4eea8ee153f1aae070fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98cb48427e84d8fa212869f63370add in term 1.
I20260929 15:57:11.610860 23195 leader_election.cc:302] T 00000000000000000000000000000000 P a98cb48427e84d8fa212869f63370add [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a98cb48427e84d8fa212869f63370add, f484056955bb4eea8ee153f1aae070fc; no voters:
I20260929 15:57:11.611305 23403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a98cb48427e84d8fa212869f63370add [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:11.618938 23403 raft_consensus.cc:686] T 00000000000000000000000000000000 P a98cb48427e84d8fa212869f63370add [term 1 LEADER]: Becoming Leader. State: Replica: a98cb48427e84d8fa212869f63370add, State: Running, Role: LEADER
I20260929 15:57:11.620301 23403 consensus_queue.cc:237] T 00000000000000000000000000000000 P a98cb48427e84d8fa212869f63370add [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98cb48427e84d8fa212869f63370add" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42789 } } peers { permanent_uuid: "9e39f91d0da842769c276bd202d468c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33081 } } peers { permanent_uuid: "f484056955bb4eea8ee153f1aae070fc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41069 } }
W20260929 15:57:11.628015 23428 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f484056955bb4eea8ee153f1aae070fc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 15:57:11.628185 23428 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 15:57:11.635294 23425 sys_catalog.cc:455] T 00000000000000000000000000000000 P a98cb48427e84d8fa212869f63370add [sys.catalog]: SysCatalogTable state changed. Reason: New leader a98cb48427e84d8fa212869f63370add. Latest consensus state: current_term: 1 leader_uuid: "a98cb48427e84d8fa212869f63370add" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98cb48427e84d8fa212869f63370add" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42789 } } peers { permanent_uuid: "9e39f91d0da842769c276bd202d468c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33081 } } peers { permanent_uuid: "f484056955bb4eea8ee153f1aae070fc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41069 } } }
I20260929 15:57:11.635818 23425 sys_catalog.cc:458] T 00000000000000000000000000000000 P a98cb48427e84d8fa212869f63370add [sys.catalog]: This master's current role is: LEADER
W20260929 15:57:11.638393 23198 tablet.cc:2369] T 00000000000000000000000000000000 P a98cb48427e84d8fa212869f63370add: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:57:11.635367 23177 master_runner.cc:186] Error getting master registration for 127.0.97.61:33081: OK, instance_id { permanent_uuid: "9e39f91d0da842769c276bd202d468c4" instance_seqno: 1790697430952458 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 15:57:11.656925 23177 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 15:57:11.659938 23420 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a98cb48427e84d8fa212869f63370add: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 15:57:11.660055 23420 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 15:57:11.660174 23433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:57:11.646487 23326 master_runner.cc:186] Error getting master registration for 127.0.97.61:33081: OK, instance_id { permanent_uuid: "9e39f91d0da842769c276bd202d468c4" instance_seqno: 1790697430952458 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 15:57:11.662266 23326 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:57:11.664415 23433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:57:11.684454 23318 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41069 } attempt: 5
Sep 29 15:57:11 dist-test-slave-r0ck krb5kdc[23164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 15:57:11.696090 23323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:11.652005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57260 (local address 127.0.97.61:33081)
0929 15:57:11.652197 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:11.652204 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:11.660669 (+ 8465us) server_negotiation.cc:407] Connection header received
0929 15:57:11.660856 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:11.660861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:11.660938 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:11.661029 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:11.677482 (+ 16453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:11.678383 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:11.679674 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:11.680353 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:11.680476 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:11.683695 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:11.683704 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:11.683710 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:11.683748 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:11.690263 (+ 6515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:11.691525 (+ 1262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:11.691531 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:11.691558 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:11.692099 (+ 541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:11.692813 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:11.692818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:11.692843 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:11.693451 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:11.693822 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:11.694899 (+ 1077us) server_negotiation.cc:299] Negotiation successful
0929 15:57:11.695320 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 15:57:11.705173 23368 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f484056955bb4eea8ee153f1aae070fc [term 1 FOLLOWER]: Refusing update from remote peer a98cb48427e84d8fa212869f63370add: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:57:11.706360 23425 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a98cb48427e84d8fa212869f63370add [LEADER]: Connected to new peer: Peer: permanent_uuid: "f484056955bb4eea8ee153f1aae070fc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:57:11.708271 23322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:11.685052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41069 (local address 127.0.0.1:41626)
0929 15:57:11.685467 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:11.685486 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:11.685581 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:11.686421 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:11.686426 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:11.686956 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:11.687140 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:11.687151 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:11.688660 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:11.688666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:11.690108 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:11.690123 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:11.691337 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:11.691344 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:11.691548 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:11.692162 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:11.692188 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:11.695003 (+ 2815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:11.705264 (+ 10261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:11.705271 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:11.705277 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:11.705811 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:11.706344 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:11.706349 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:11.706352 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:11.706524 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:11.707232 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:11.707239 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:11.707420 (+ 181us) client_negotiation.cc:769] Sending connection context
0929 15:57:11.707565 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 15:57:11.707851 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267}
I20260929 15:57:11.709626 23396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:11.685777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41626 (local address 127.0.97.60:41069)
0929 15:57:11.685971 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:11.685976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:11.685993 (+ 17us) server_negotiation.cc:407] Connection header received
0929 15:57:11.686040 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:11.686045 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:11.686112 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:11.686204 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:11.687354 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:11.688449 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:11.690286 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:11.690979 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:11.691107 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:11.695204 (+ 4097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:11.695211 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:11.695215 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:11.695261 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:11.705093 (+ 9832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:11.705979 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:11.705987 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:11.706000 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:11.706200 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:11.706675 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:11.706680 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:11.706683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:11.707067 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:11.708525 (+ 1458us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:11.708680 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 15:57:11.709163 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 15:57:11.712605 23397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:11.638947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58664 (local address 127.0.97.62:42789)
0929 15:57:11.639107 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:11.639113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:11.648901 (+ 9788us) server_negotiation.cc:407] Connection header received
0929 15:57:11.649177 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:11.649183 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:11.649284 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:11.649371 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:11.661336 (+ 11965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:11.662591 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:11.674448 (+ 11857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:11.675396 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:11.675530 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:11.695093 (+ 19563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:11.695103 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:11.695108 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:11.695155 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:11.703255 (+ 8100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:11.710226 (+ 6971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:11.710232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:11.710235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:11.710406 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:11.710865 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:11.710872 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:11.710878 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:11.711249 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:11.711488 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:11.712028 (+ 540us) server_negotiation.cc:299] Negotiation successful
0929 15:57:11.712250 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 15:57:11.714474 23399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:11.661789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41610 (local address 127.0.97.60:41069)
0929 15:57:11.662021 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:11.662028 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:11.678308 (+ 16280us) server_negotiation.cc:407] Connection header received
0929 15:57:11.678972 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:11.678978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:11.679072 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:11.679145 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:11.698030 (+ 18885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:11.699145 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:11.701042 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:11.701744 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:11.701856 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:11.705086 (+ 3230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:11.705092 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:11.705098 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:11.705132 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:11.708982 (+ 3850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:11.710764 (+ 1782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:11.710772 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:11.710778 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:11.710944 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:11.711384 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:11.711389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:11.711392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:11.711719 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:11.711973 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:11.712487 (+ 514us) server_negotiation.cc:299] Negotiation successful
0929 15:57:11.712782 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 15:57:11.717044 23318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e39f91d0da842769c276bd202d468c4: Bootstrap starting.
I20260929 15:57:11.725198 23318 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e39f91d0da842769c276bd202d468c4: Neither blocks nor log segments found. Creating new log.
W20260929 15:57:11.725919 23347 tablet.cc:2369] T 00000000000000000000000000000000 P f484056955bb4eea8ee153f1aae070fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:57:11.727402 23318 log.cc:824] T 00000000000000000000000000000000 P 9e39f91d0da842769c276bd202d468c4: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:11.739287 23402 sys_catalog.cc:455] T 00000000000000000000000000000000 P f484056955bb4eea8ee153f1aae070fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader a98cb48427e84d8fa212869f63370add. Latest consensus state: current_term: 1 leader_uuid: "a98cb48427e84d8fa212869f63370add" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98cb48427e84d8fa212869f63370add" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42789 } } peers { permanent_uuid: "9e39f91d0da842769c276bd202d468c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33081 } } peers { permanent_uuid: "f484056955bb4eea8ee153f1aae070fc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41069 } } }
I20260929 15:57:11.739570 23402 sys_catalog.cc:458] T 00000000000000000000000000000000 P f484056955bb4eea8ee153f1aae070fc [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:57:11.741384 23318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e39f91d0da842769c276bd202d468c4: No bootstrap required, opened a new log
I20260929 15:57:11.746332 23438 mvcc.cc:205] Tried to move back new op lower bound from 7334696680250015744 to 7334696679925583872. Current Snapshot: MvccSnapshot[applied={T|T < 7334696680250015744}]
I20260929 15:57:11.747884 23439 mvcc.cc:205] Tried to move back new op lower bound from 7334696680250015744 to 7334696679925583872. Current Snapshot: MvccSnapshot[applied={T|T < 7334696680250015744}]
I20260929 15:57:11.754509 23402 sys_catalog.cc:455] T 00000000000000000000000000000000 P f484056955bb4eea8ee153f1aae070fc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a98cb48427e84d8fa212869f63370add" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98cb48427e84d8fa212869f63370add" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42789 } } peers { permanent_uuid: "9e39f91d0da842769c276bd202d468c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33081 } } peers { permanent_uuid: "f484056955bb4eea8ee153f1aae070fc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41069 } } }
I20260929 15:57:11.754755 23402 sys_catalog.cc:458] T 00000000000000000000000000000000 P f484056955bb4eea8ee153f1aae070fc [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:57:11.765291 23425 sys_catalog.cc:455] T 00000000000000000000000000000000 P a98cb48427e84d8fa212869f63370add [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a98cb48427e84d8fa212869f63370add" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98cb48427e84d8fa212869f63370add" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42789 } } peers { permanent_uuid: "9e39f91d0da842769c276bd202d468c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33081 } } peers { permanent_uuid: "f484056955bb4eea8ee153f1aae070fc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41069 } } }
I20260929 15:57:11.765600 23425 sys_catalog.cc:458] T 00000000000000000000000000000000 P a98cb48427e84d8fa212869f63370add [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:11.766487 23433 catalog_manager.cc:1347] Generated new cluster ID: 35713b37595340e092ac3b3f79f5bb04
I20260929 15:57:11.766579 23433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:57:11.767114 23318 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e39f91d0da842769c276bd202d468c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98cb48427e84d8fa212869f63370add" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42789 } } peers { permanent_uuid: "9e39f91d0da842769c276bd202d468c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33081 } } peers { permanent_uuid: "f484056955bb4eea8ee153f1aae070fc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41069 } }
I20260929 15:57:11.767544 23318 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e39f91d0da842769c276bd202d468c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e39f91d0da842769c276bd202d468c4, State: Initialized, Role: FOLLOWER
I20260929 15:57:11.768105 23318 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e39f91d0da842769c276bd202d468c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98cb48427e84d8fa212869f63370add" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42789 } } peers { permanent_uuid: "9e39f91d0da842769c276bd202d468c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33081 } } peers { permanent_uuid: "f484056955bb4eea8ee153f1aae070fc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41069 } }
I20260929 15:57:11.769330 23444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e39f91d0da842769c276bd202d468c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98cb48427e84d8fa212869f63370add" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42789 } } peers { permanent_uuid: "9e39f91d0da842769c276bd202d468c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33081 } } peers { permanent_uuid: "f484056955bb4eea8ee153f1aae070fc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41069 } } }
I20260929 15:57:11.769986 23444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e39f91d0da842769c276bd202d468c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:57:11.770655 23318 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e39f91d0da842769c276bd202d468c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:57:11.770684 23293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a98cb48427e84d8fa212869f63370add" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9e39f91d0da842769c276bd202d468c4"
I20260929 15:57:11.771055 23292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e39f91d0da842769c276bd202d468c4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:11.777825 23292 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9e39f91d0da842769c276bd202d468c4 [term 1 FOLLOWER]: Refusing update from remote peer a98cb48427e84d8fa212869f63370add: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:57:11.781499 23293 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9e39f91d0da842769c276bd202d468c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98cb48427e84d8fa212869f63370add in term 1.
I20260929 15:57:11.786500 23403 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a98cb48427e84d8fa212869f63370add [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e39f91d0da842769c276bd202d468c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:57:11.826330 23444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e39f91d0da842769c276bd202d468c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a98cb48427e84d8fa212869f63370add. Latest consensus state: current_term: 1 leader_uuid: "a98cb48427e84d8fa212869f63370add" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98cb48427e84d8fa212869f63370add" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42789 } } peers { permanent_uuid: "9e39f91d0da842769c276bd202d468c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33081 } } peers { permanent_uuid: "f484056955bb4eea8ee153f1aae070fc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41069 } } }
I20260929 15:57:11.826293 23453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e39f91d0da842769c276bd202d468c4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a98cb48427e84d8fa212869f63370add" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98cb48427e84d8fa212869f63370add" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42789 } } peers { permanent_uuid: "9e39f91d0da842769c276bd202d468c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33081 } } peers { permanent_uuid: "f484056955bb4eea8ee153f1aae070fc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41069 } } }
I20260929 15:57:11.826722 23444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e39f91d0da842769c276bd202d468c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:57:11.826777 23453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e39f91d0da842769c276bd202d468c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:57:11.827024 23433 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:57:11.827174 23425 sys_catalog.cc:455] T 00000000000000000000000000000000 P a98cb48427e84d8fa212869f63370add [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a98cb48427e84d8fa212869f63370add" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98cb48427e84d8fa212869f63370add" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42789 } } peers { permanent_uuid: "9e39f91d0da842769c276bd202d468c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33081 } } peers { permanent_uuid: "f484056955bb4eea8ee153f1aae070fc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41069 } } }
I20260929 15:57:11.827438 23425 sys_catalog.cc:458] T 00000000000000000000000000000000 P a98cb48427e84d8fa212869f63370add [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:11.829093 23433 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:57:11.853518 23433 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a98cb48427e84d8fa212869f63370add: Generated new TSK 0
I20260929 15:57:11.855261 23433 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:57:11.899489 23460 catalog_manager.cc:1259] Loaded cluster ID: 35713b37595340e092ac3b3f79f5bb04
I20260929 15:57:11.899657 23460 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9e39f91d0da842769c276bd202d468c4: loading cluster ID for follower catalog manager: success
I20260929 15:57:11.908545 23460 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9e39f91d0da842769c276bd202d468c4: acquiring CA information for follower catalog manager: success
I20260929 15:57:11.911103 23460 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9e39f91d0da842769c276bd202d468c4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 15:57:11.972780 23397 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:57:11.966723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58674 (local address 127.0.97.62:42789)
0929 15:57:11.966931 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:11.966939 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:11.970422 (+ 3483us) server_negotiation.cc:407] Connection header received
0929 15:57:11.970779 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:11.970786 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:11.970899 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:11.970986 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:11.972478 (+ 1492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58674: BlockingRecv error: recv got EOF from 127.0.0.1:58674 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 15:57:11.981088 23399 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:57:11.978562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41632 (local address 127.0.97.60:41069)
0929 15:57:11.978798 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:11.978805 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:11.979215 (+ 410us) server_negotiation.cc:407] Connection header received
0929 15:57:11.979483 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:11.979490 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:11.979571 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:11.979643 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:11.980804 (+ 1161us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41632: BlockingRecv error: recv got EOF from 127.0.0.1:41632 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260929 15:57:11.986377 23323 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:57:11.974095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57276 (local address 127.0.97.61:33081)
0929 15:57:11.974373 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:11.974382 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:11.974411 (+ 29us) server_negotiation.cc:407] Connection header received
0929 15:57:11.974509 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:11.974518 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:11.974669 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:11.974846 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:11.986145 (+ 11299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57276: BlockingRecv error: recv got EOF from 127.0.0.1:57276 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 15:57:12.035250 23323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:12.019058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57286 (local address 127.0.97.61:33081)
0929 15:57:12.019318 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:12.019326 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:12.027430 (+ 8104us) server_negotiation.cc:407] Connection header received
0929 15:57:12.028002 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:12.028008 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:12.028265 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:12.028407 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:12.029121 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:12.030309 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:12.032310 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:12.032980 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:12.033083 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:12.033846 (+ 763us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:12.034442 (+ 596us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:12.034562 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:12.034809 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 15:57:12.034903 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260929 15:57:12.076854 23399 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:57:12.068620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41648 (local address 127.0.97.60:41069)
0929 15:57:12.068837 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:12.068844 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:12.074741 (+ 5897us) server_negotiation.cc:407] Connection header received
0929 15:57:12.075265 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:12.075276 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:12.075401 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:12.075484 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:12.076600 (+ 1116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41648: BlockingRecv error: recv got EOF from 127.0.0.1:41648 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 15:57:12.123056 23323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:12.108230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57296 (local address 127.0.97.61:33081)
0929 15:57:12.108417 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:12.108423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:12.115720 (+ 7297us) server_negotiation.cc:407] Connection header received
0929 15:57:12.116240 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:12.116247 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:12.116342 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:12.116407 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:12.117088 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:12.118281 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:12.120620 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:12.121448 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:12.121559 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:12.122194 (+ 635us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:12.122363 (+ 169us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:12.122431 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:12.122665 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 15:57:12.122751 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260929 15:57:12.168483 23399 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:57:12.158109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41660 (local address 127.0.97.60:41069)
0929 15:57:12.158379 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:12.158386 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:12.166427 (+ 8041us) server_negotiation.cc:407] Connection header received
0929 15:57:12.166778 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:12.166785 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:12.166901 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:12.166992 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:12.168201 (+ 1209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41660: BlockingRecv error: recv got EOF from 127.0.0.1:41660 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 15:57:12.217864 23323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:12.202811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57304 (local address 127.0.97.61:33081)
0929 15:57:12.203011 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:12.203018 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:12.211036 (+ 8018us) server_negotiation.cc:407] Connection header received
0929 15:57:12.211521 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:12.211527 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:12.211620 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:12.211690 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:12.212358 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:12.213504 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:12.215522 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:12.216227 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:12.216325 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:12.216938 (+ 613us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:12.217070 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:12.217133 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:12.217435 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 15:57:12.217532 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260929 15:57:12.262658 23399 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:57:12.252546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41662 (local address 127.0.97.60:41069)
0929 15:57:12.252753 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:12.252761 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:12.260462 (+ 7701us) server_negotiation.cc:407] Connection header received
0929 15:57:12.260999 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:12.261010 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:12.261138 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:12.261252 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:12.262396 (+ 1144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41662: BlockingRecv error: recv got EOF from 127.0.0.1:41662 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 15:57:12.309556 23323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:12.298151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57312 (local address 127.0.97.61:33081)
0929 15:57:12.298390 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:12.298396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:12.302059 (+ 3663us) server_negotiation.cc:407] Connection header received
0929 15:57:12.302316 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:12.302324 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:12.302644 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:12.302792 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:12.303516 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:12.304787 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:12.307317 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:12.308073 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:12.308226 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:12.308727 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:12.308864 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:12.308966 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:12.309137 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 15:57:12.309235 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
W20260929 15:57:12.321781 23399 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:57:12.319625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41664 (local address 127.0.97.60:41069)
0929 15:57:12.319880 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:12.319886 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:12.319909 (+ 23us) server_negotiation.cc:407] Connection header received
0929 15:57:12.320109 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:12.320117 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:12.320235 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:12.320311 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:12.321482 (+ 1171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41664: BlockingRecv error: recv got EOF from 127.0.0.1:41664 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
I20260929 15:57:12.342242 23323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:12.334831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57322 (local address 127.0.97.61:33081)
0929 15:57:12.335022 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:12.335028 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:12.335623 (+ 595us) server_negotiation.cc:407] Connection header received
0929 15:57:12.335845 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:12.335852 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:12.335958 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:12.336075 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:12.336528 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:12.337741 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:12.339724 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:12.340430 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:12.340576 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:12.341247 (+ 671us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:12.341451 (+ 204us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:12.341555 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:12.341804 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 15:57:12.341892 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 15:57:12.355629 23399 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:57:12.351446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41678 (local address 127.0.97.60:41069)
0929 15:57:12.351645 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:12.351651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:12.353985 (+ 2334us) server_negotiation.cc:407] Connection header received
0929 15:57:12.354208 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:12.354214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:12.354324 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:12.354445 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:12.355394 (+ 949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41678: BlockingRecv error: recv got EOF from 127.0.0.1:41678 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 15:57:12.386806 23323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:12.379028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57336 (local address 127.0.97.61:33081)
0929 15:57:12.379221 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:12.379228 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:12.379787 (+ 559us) server_negotiation.cc:407] Connection header received
0929 15:57:12.380131 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:12.380138 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:12.380254 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:12.380378 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:12.380842 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:12.382061 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:12.384119 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:12.384985 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:12.385181 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:12.385756 (+ 575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:12.386064 (+ 308us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:12.386189 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:12.386374 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 15:57:12.386481 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260929 15:57:12.396541 23399 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:57:12.394105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41682 (local address 127.0.97.60:41069)
0929 15:57:12.394285 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:12.394291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:12.394813 (+ 522us) server_negotiation.cc:407] Connection header received
0929 15:57:12.395013 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:12.395020 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:12.395121 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:12.395239 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:12.396272 (+ 1033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41682: BlockingRecv error: recv got EOF from 127.0.0.1:41682 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 15:57:12.443104 23323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:12.435967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57344 (local address 127.0.97.61:33081)
0929 15:57:12.436179 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:12.436186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:12.436550 (+ 364us) server_negotiation.cc:407] Connection header received
0929 15:57:12.436726 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:12.436733 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:12.436879 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:12.437005 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:12.437436 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:12.438577 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:12.440347 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:12.441393 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:12.441611 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:12.442042 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:12.442228 (+ 186us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:12.442332 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:12.442620 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 15:57:12.442709 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260929 15:57:12.452737 23399 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:57:12.450775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41684 (local address 127.0.97.60:41069)
0929 15:57:12.450971 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:12.450977 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:12.451292 (+ 315us) server_negotiation.cc:407] Connection header received
0929 15:57:12.451468 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:12.451474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:12.451563 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:12.451671 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:12.452502 (+ 831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41684: BlockingRecv error: recv got EOF from 127.0.0.1:41684 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 15:57:12.527191 23323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:12.521766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57358 (local address 127.0.97.61:33081)
0929 15:57:12.522002 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:12.522007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:12.522275 (+ 268us) server_negotiation.cc:407] Connection header received
0929 15:57:12.522374 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:12.522378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:12.522455 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:12.522548 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:12.522961 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:12.523711 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:12.525287 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:12.525882 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:12.526031 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:12.526475 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:12.526579 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:12.526658 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:12.526837 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 15:57:12.526927 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260929 15:57:12.535187 23399 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:57:12.533089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41694 (local address 127.0.97.60:41069)
0929 15:57:12.533464 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:12.533476 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:12.533966 (+ 490us) server_negotiation.cc:407] Connection header received
0929 15:57:12.534038 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:12.534042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:12.534113 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:12.534199 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:12.534934 (+ 735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41694: BlockingRecv error: recv got EOF from 127.0.0.1:41694 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
I20260929 15:57:12.630144 23428 catalog_manager.cc:1259] Loaded cluster ID: 35713b37595340e092ac3b3f79f5bb04
I20260929 15:57:12.630236 23428 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f484056955bb4eea8ee153f1aae070fc: loading cluster ID for follower catalog manager: success
I20260929 15:57:12.632613 23428 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f484056955bb4eea8ee153f1aae070fc: acquiring CA information for follower catalog manager: success
I20260929 15:57:12.634356 23428 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f484056955bb4eea8ee153f1aae070fc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 15:57:12.673672 23323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:12.666568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57360 (local address 127.0.97.61:33081)
0929 15:57:12.666777 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:12.666786 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:12.667332 (+ 546us) server_negotiation.cc:407] Connection header received
0929 15:57:12.667479 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:12.667486 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:12.667601 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:12.667805 (+ 204us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:12.668180 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:12.669502 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:12.670848 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:12.671731 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:12.671886 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:12.672669 (+ 783us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:12.672829 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:12.672939 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:12.673149 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 15:57:12.673245 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 15:57:12.689234 23399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:12.681222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41696 (local address 127.0.97.60:41069)
0929 15:57:12.681703 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:12.681713 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:12.681906 (+ 193us) server_negotiation.cc:407] Connection header received
0929 15:57:12.682023 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:12.682030 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:12.682308 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:12.682474 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:12.683079 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:12.684732 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:12.686189 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:12.687024 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:12.687178 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:12.687730 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:12.688511 (+ 781us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:12.688678 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:12.688738 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 15:57:12.688842 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280}
I20260929 15:57:12.709759 23323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:12.700180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57362 (local address 127.0.97.61:33081)
0929 15:57:12.700384 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:12.700392 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:12.700416 (+ 24us) server_negotiation.cc:407] Connection header received
0929 15:57:12.700719 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:12.700727 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:12.700815 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:12.700893 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:12.701905 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:12.703428 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:12.706780 (+ 3352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:12.707711 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:12.707834 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:12.708661 (+ 827us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:12.708870 (+ 209us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:12.708944 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:12.709204 (+ 260us) server_negotiation.cc:299] Negotiation successful
0929 15:57:12.709315 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 15:57:12.710256 23565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:12.698347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58676 (local address 127.0.97.62:42789)
0929 15:57:12.699101 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:12.699109 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:12.699655 (+ 546us) server_negotiation.cc:407] Connection header received
0929 15:57:12.700214 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:12.700225 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:12.700604 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:12.700830 (+ 226us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:12.702157 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:12.703744 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:12.706833 (+ 3089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:12.707737 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:12.707868 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:12.708877 (+ 1009us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:12.709405 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:12.709495 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:12.709759 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 15:57:12.709869 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":342,"threads_started":1}
I20260929 15:57:12.716425 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 23177
I20260929 15:57:12.716668 23399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:12.701189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41712 (local address 127.0.97.60:41069)
0929 15:57:12.701408 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:12.701415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:12.710519 (+ 9104us) server_negotiation.cc:407] Connection header received
0929 15:57:12.710848 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:12.710857 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:12.710979 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:12.711057 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:12.711700 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:12.713012 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:12.714777 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:12.715341 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:12.715469 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:12.715845 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:12.715985 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:12.716090 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:12.716238 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 15:57:12.716312 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 15:57:12.717494 23239 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:12.836547 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 23177
W20260929 15:57:12.857992 23345 connection.cc:535] server connection from 127.0.0.1:41604 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 15:57:12.858735 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 23251
I20260929 15:57:12.860286 23313 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:12.975178 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 23251
I20260929 15:57:12.999961 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskorfijW/build/asan/bin/kudu pid 23326
I20260929 15:57:13.001415 23388 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:13.113104 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskorfijW/build/asan/bin/kudu with pid 23326
2026-09-29T15:57:13Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4534 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (8992 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (357408 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST